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ABSTRACT 

A study of the "efficiency and efficacy 11 of 
evaluation practices of the Illinois Division of Vocational and 
Technical Education as conducted by Educational Testing Service is 
reported on. The study was directed toward understanding how 
evaluation practices of the Division, with. largest emphasis upon the 
performance of the Three Phar^ System used to conduct statewide 
Evaluation of occupational pisXjrams, might be improved over 
successive months and years. Analysis of the System was conducted by 
use of several methodologies: (1) interviews of Division personnel, 
(2) sampling of agencies for study, (3) collection of data from 
record systems of the Division, (4) submission of questionnaires to 
piahners and team members, and (5) independent study of trends in 
statewide evaluation. 'Outcomes of the Three Phase System suggest it 
is well received by local education agencies and by the team members 
who have contributed to it. However, it has numerous deficiencies 
most of which arise from the fact that it does not emphasize or 
produce product information. Among the favorable features of the 
System is its self-reactance. Long and short range recommendations 
for changes in system-pfcmning and functioning conclude this report. 
(Author/CK) 
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: The State of Illinois Advisory Council 6ri Vocational Education is 
charged by federal legislation with the responsib for conducting 

an' ^independent evaluation of the effectiveness of vocational education 
pragfamsi servicesi and activities /^t^n/^e.--s£||e.. Specifically 
^l&s^r^sponsibliity entails -an intensive study of some of the evaluation 
effof|s of the bivision of Vocational arid 1 T|^nical :td ; N lf£i6m 

This document reports a; study of tft& "efficiency of 
evaluation of the Iliifipis, piyision of Vocat^ 

Education as condjictedjiby fEducStipnalf Tes tjgg | #By ice, otebehal f of ihe 
My||ory Councils Readers wishing |^ first learn about its findings and 
Cfcott shpul| turn direc tly to the. sections Outcomes aridf Recdmf 

mendatibhS t op pages 29^ and 73 , i^spec tiy ely*. f Wejpv ec|ding|fs^|io^ 
describe the context in which the Division f u^tipnsj,, the Three Fhase 



The study was directed £<waf4^^ 
tices of the Divi¥Iqn7 wit;fr largest; the «, 

T|ree Phase System -used to conduct statewiS^ of occupational 

pfogfamst fiight be improved over successive months arid years* The three 
phlses of the study» which overlapped in their conduct^ were desigh|td to 
anWef these questions:, (1) What infpraatiori Vhpulcl evaluation p^ 
-C?J?Hpw-weli are current eyaluafelpn .pf actices meeting these ihfpfmatipn 
fteedsT and- (3) How might eyalMtiph prac ti^ 

^alysis^of^the System was conducted by use of several methodol- 
ogies: (1) interviews of DiyisiSti per (2) f|ipling of agencies 
fo^istudy, (3) coilectipn of data from record systems of the Division, 
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(4) submission of questionnaires to planners and team members, ahd ^f) 
independent study of trends in statewide evaluation. 

Outcomes of tfie study jof 'the Tfee&-Phas^ suggest it is? 

well received; by -local, education by the team members \ whp^ 

Have, cdnj&ifated to it. However, it Mas rcurn^ most^gf 
which arise \ from the fact thai it does r& .o^^pducerpip^ict 
information^ The f»erfomajSces df git iuates of occupational programs- 
are hot assessed;, without this infora thev ^ 

System ^ar.et res trictf 3 in> their $ff ectiyeness. ^sdy the Syst^ doesr 
not provide a i corner etiensiye f ramwo^ 

Mong ;the f ayoM 
That is* it Ms t been srt inf <^ 

to improve; i t can ^bte^rl^ulaf ly: :coMetted ;aS| ilapplled By^u^ing , this? , 
valuable f eature, it must be fojpi^ 

cliju^ed in- the di^a^ tpi^jand- sJSort 

range recommendations for' changes in and* f unc : tionin|| 

con^clude this fepMDr|> '7 "~- [- % 
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INTRODUCTION 

Background of Study 

Vocational and technical education programs are an integral part of 
the total education package that taxpayers are asked to support* There- 
fore, these programs must be relevant to needs and the use of program funds 
must be accounted for* To accomplish this task, the State of Illinois Ad- 
visory Council on Vocational Education is charged by federal legislation 
with responsibility for conducting an independent evaluation of the effective- 
ness of vocational education programs, services, and activities within the 
state. Specifically, this responsibility entails intensive study of the 
evaluation efforts of the Illinois Division of Vocational and Technical 
Education (DVTE) ♦ 

This study was funded by the Illinois Advisory Council* It is an inde- 
pendent investigation of the evaluation practices of the DVTE, having a pri- 
mary focus on the performance of the Three Phase System for Statewide Eval- 
uation of. Occupational Education Programs administered by the Program 
Approval and Evaluation Unit of the Division* The Advisory Council requested 
that it be furnished with data and action recommendations concerning the - 
"efficiency and efficacy" of the Division 1 s range of evaluation activities. 
It was particularly interested in determining if the evaluative efforts of 
the Division have been appropriate and if they have related well to the needs 
of its programs and their sponsors. 

Illinois Division of Vocational and Technical Education 
Organization 

The Division is organized under the Board of Vocational Education and 
Rehabilitation through which it is related to the Governors Advisory Council. 



It possesses a director and two assistant directors having responsibility 
for the performance of eight operational units. An Administrative Planning 
Council which is made up of th« eight Unit Coordinators relates to the Office 
of the Director. (See Figure A.) 

The Superintendent of Public Instruction is designated by statute to 
be the Executive Officer for work of the Division: 

"The Superintendent, of Public Instruction shall be the Executive 
Officer for all of its work pertaining to Vocational and Technical 
Education, with responsibility to represent the Board in the admin- 
istration of its plans and programs for vocational and technical 
education." (Chap. 122, Sec. 695 I 2, Illinois Revised Statutes, 
1967.) 

Evaluation System 

The 1960*8 was a decade of great emphasis on vocational education. Par- 
ticularly important toward the general upgrading of this nation's vocational 
skills was the Vocational Education Act of 1963 and the subsequent Amendments 
of 1968. The need for state and local evaluation for the purpose of deter- 
mining whether the intentions of the legislature are met was specifically de- 
tailed in the 1968 Amendments. In the attempt to comply with the instructions 
that evaluation of programs be accomplished, the Illinois Division*of Voca- 
tional and Technical Education developed a comprehensive plan referred to as 
the Three Phase System for Statewide Evaluation of Occupational Educational 
Programs (Bulletin 35-971).' 

The cnordinated state system for evaluation of vocational programs that 
has been developed has incorporated within it the important tasks of account- 
ability, planning, and improvement of programs; the goals of this system are 
as follows; 

*Three Phase Sy stem for Statewide Evaluation of Occupational Education 
Programs. Springfield, Illinois: Board of Vocational Education and Rehabil- 
itation; Division of Vocational and Technical Education, 1971. 
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1. to promote and assist with development of quality 
occupational education programs at the local level; 

2. to provide the Division of Vocational and Technical 
Education with data' upon which better statewide plan- 
ning, of occupational education programs can be based: 
and 

3. to help assure accountability of Federal and State 
funds allocated to local occupational education pro- 
grams* 2 

Following from these goals are established objectives for occupational 
education in Illinois. A number of first and second priority objectives have 
been established, : * 



The purpose of the Three Phase System of Evaluation yas to determine; .the 

extent to which the objectives have been and , are being met.- The functions <of 

•* *- * - - 

each of the phases are: 



Phase I Utiiizatioti of self ^evaluation in program planning 
by the local .district: culminating in the develops 
ment of a One and Five Year Plair. 



Phase II Program approval -by staff of the Illinois Division 
of Vocational and Technical Education. 

Phase III External evaluation by a visiting team to develop 
a profile of the local program with conclusions 
and recommendations to the local district. 



The State Plan for the Administration of Vocational and Technical 
Education in Illinois - 1972 , prepared by the Division of Vocational and 
Technical Education ip cooperation with several other agencies, indicates that 
the Division's top priority during Fiscal Year i972 was to be full implemen- 
tation of Phase III. 3 The independent evaluation activities of the Educa- 
tional Testing Service was directed toward a consideration of the efficiency 
and efficacy of the Illinois Evaluation system as it relates to this document. 



2 Ibid , p. 2. 

3 A State Plan for the Administration of Vocational and Technical Education . 
Spr ingfield, Illinois: Board of Vocational Education and Rehabilitation, 
Division of Vocational and Technical Education, 1971. 
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Description of the Three Phase System 

Phases I and II are annual processes* while Phase III occurs every two 
to live years or as warranted. The following sections provide a brief des- 
cription of the forms, procedures, and schedules of the Three Phase System. 
This description is intended to enable persons not familiar with it to under- 
stand (1) how, during Phase I, people in local education agencies (LEA f s) re- 
ceived descriptions of the System and guidelines i jr the preparation of plan- 
ning documents; (2) how, during Phase II, LEA plans were reviewed for approval; 
(3): how, during Phase III, site visits were conducted; and (4) how the program 
planning and evaluation unit undertook to evaluate the effectiveness of its 
sysfceu. Figure B;, which follows, presents a Schematic of the Three Phase 
System together with a figure showing^the-relatioriship of Division and 
local personnel as the System is implemented. 
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Phase„I: Development of Local Plans 

the Illinois Three Phase System, as it was used during Fiscal Year .1972, 
was developed over several years. Some of the documents that are no\r essen- 
tial parts of the System (on-site evaluation instruments, for instance) were 
systematically developed from draft devices through field trials, modification, 
additional trials and further revisions. Others were not formal!*' developed in 
the same way, but all have gradually been woven together until, .t present, 
they link with each other in many ways. During Phase I, the LEA must be able 
to read and understand, or receive help to understand, two basic documents. 
One is titled Guidelines and Format for Preparing Local District One and Five 
Year Plan for Vocational and Technical Education. k This document includes four 
pages of narrative intended to help the writers of plans and is followed by ten 
pages of forms to be completed by the agency as part of its application process. 
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^Guidelines and Format Planning for Preparing Local District One and Five 
Year Plan for Vocational and Technical Education . Bulletin No. 2-971, Spring- 
field: State of Illinois, Division of Vocational and Technical Education. 
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Schematic of Three Phase System for 
Statewide Evaluation of Occupational Education Programs 
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Phase I: Relationship of Division and Local Personnel 

Figure B 
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The second document of importance for planning reference is titled Cri^ 
teria for Program Approval and Financial Support , 5 This fifteen page docu- 
ment describes nine types of general provisions (A-I) to be met for all levels 
of planning and also describes the five types of instructional programs (A-E) 
for which support may be granted.* It gives minimum requirements for person- 
nel and tells how funds are allocated- Fund allocations go toward basic claims 
and for six additional factors that augment basic claims. The additional fac- 
tors are: 

Factor I - Relative Ability to Pay 

Factor II - Provisions for Educating Disadvantaged Students 
Factor III - Special Organizations 
Factor IV - Initial Programs 
Factor V - Manpower Priorities 
Factor VI - Programs for Handicapped 

Phase I and its later companion phases are described in a document titled 
Three Phase System for Statewide Evaluation of Occupational Programs. 6 This 
30-page document enables its readers to place the planning phase in its in- 
tended relationship with the approval and on-site visitation phases. It pro- 
vides, with its emphasis on visitation, a very useful orientation for those 
having the task of developing LEA plans. 

Phase II: Approval of Local Plans 

The local plan, when it has been approved by the district and its local 
governing board, is transmitted through the Superintendent of Educational Serv- 
ice Region or the Illinois Junior College Board .to the Program Approval and 



5 Criteria for Program Approval and Financial Support , Bulletin No. 4-171-2 
Springfield: State of Illinois, Division of Vocational and Technical Education. 

*Division Bulletins having the same Bulletin Number do not always have the 
same contents.. Small changes in the contents of bulletins are made at regular 
intervals without changes being made in the numbering system or the insertion • 
of dates whereby their users can readily distinguish between them. 

6 Three Phase System for Statewide Evaluation of Occupational Education 
Programs , op . cit . 
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Evaluation Unit of the Division. This unit, with assistance from other per- 
sons, evaluates the Local Plan and recommends that it be given one of three 
approval levels: Fully Approved (A), Conditionally Approved (CA), and Not _^ 
Approved (NA). 

A large fraction of the work involved in reviewing local plans for ap- 
proval is carried on by the eight regional directors employed by the Unit. 
They are expected to follow a 5-step procedure when reviewing the Plan: 

1. Look' at the preceding year's Plan if there was one. 

2. Read the site visitation evaluation report if there is one* 

3. Read the new Plan. 

4. Read the "suggested action" column in the evaluation report. 

5. Indicate on the report the items that are met on the Work Sheet. 

The Work Sheet is a two sided document having spaces in which to rate differ- 
ent characteristics of the Plan. It is divided into the same sections that 
must be completed for the submission of a Plan, but does not have categories 
for rating that require highly objective reading and review. 

When the review process has been completed, a Status of ^lan^Report is 
prepared on which is recorded the approval status of the plan, the approval 
of courses within the plan, and review comments. This record, a copy of which 
is appended, becomes the official document for communication about Status of 
Plan between the Division and LEA's. A master control record for local plans 
is maintained in the Unit on which the approval status and other information 
are retained.* 

Phase III: On-Site Visitations 

Phase III is, in most respects, the critical core of the Three Phase 
System. Agencies receive an extensive description of it before they prepare 
their local plan and an extensive effort is made to assure that they will not 



*See Appendix for examples of these forms. 
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fail to understand its importance. 

Agencies to be visited are not selected on a purely random basis. After 
the selection criteria that are used at a given time have been applied, an 
agency to be visited is notified. This notification sets in motion a series 
of activities that culminate in the mailing of an evaluation report to the 
agency that has been visited. Agency staff members are oriented to the visi- 
tation process through use of a film strip specifically prepared for the pur- 
pose. Prior to visitation, they are asked to complete and transmit a School 
and Community Demographic Data Form for use by the visiting team. In addition 
two questionnaire forms have been prepared for distribution to instructors, 
counselors, and school adminstrators (Professional Staff Form) and students 
(Student Form). These three documents are used to collect information about 
the agency to be visited. A report that summarizes data collected by the 
use of these forms is sent to team members prior to the evaluation visit. 

Team members are selected through application of criteria reported in 
Bulletin 35-771 which calls for team sizes varying from 3 to 20 or more mem- 
bers; each team is intended, if it is large, to include educators, business, 
and industry personnel, and former occupational students. .Single or joint 
team leaders coordinate the activities of the team on sxte. 

Two additional documents have been prepared for use by team members. 
They are (1) a Team Member Handbook for Evaluation 8 containing an overview 
plus guidelines and spaces in which to make notes, and (2) an Evaluation Team 
Member Interview Handbook 9 in which to record information assembled in the 

7 Ibid ., p. 13 . 

8 Team Member Handbook for Evaluation , Springfield: State of Illinois, 
Board of Vocational Education and Rehabilitation. 

Evaluation Team Member Interview Handbook . Springfield: State of 
Illinois, Board of Vocational Education and Technical Education . 



11 



course of personal on-site interviewing. 

Correspondence between the Division and agencies to be visited, 
records of completion of the visit, transmission of follow-up letters, and 
notati6n of the mailing of the completed report and submission of travel 
vouchers for team leaders are maintained on an Evaluation Log card in the 
Program Planning and Evaluation Unit. 

Internal Evaluation of the Three Phase System 

After an agency has been visited, four instruments, each of which 
contains from 21 to 41 items, are used to evaluate the System itself. They 
are designed for distribution to (1) teachers, (2) administrators, (3) team 
leaders, and (4) team members. Responses to the items are, when received, 
processed for use in the modification of the Three Phase System itself and 
for inclusion in the composite evaluation report prepared by the Unit on 
behalf of the Division. 10 



10 The first report prepared as a result of data collected from the Three 
Phase Evaluation System is titled Composite Evaluation Report for Occupational 
Education in the State of Illinois, Fiscal Year 1972. 
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INFORMATION NEEDS OF THE DIVISION OF 
VOCATIONAL AND TECHNICAL EDUCATION 



Users of Division Information 

The Illinois Division of Vocational and Technical Education annually 
prepares two major documents for external audiences. One is a composite 
of information assembled on forms provided by the Department of Health, 
Education and Welfare; the second is an annual descriptive repo* f Voca- 
tional Education in Illinois which is submitted to the United State Office 
of Education. The latter document tends also to be distributed in an au- 
thoritative summary for use by the Governor, the Board of Vocational Educa- 
tion and Rehabilitation of the State of Illinois, and other agencies and 
individuals. 

The Division also produces many thousands of copies of other forms 
and reports among which are requests for proposals to do funded work, proce- 
dural guidelines for the submission of requests for funds and for reimburse- 
ment of programs, and program and project descriptions. Some reports are of 
general informational value; others are for use by very limited audiences. 

An information system, or. an evaluation system which is to generate in- 
formation cannot be expected to provide all types of data for all types of 
reports and procedural documents. *~~However, it must be anticipated that a 
comprehensive information or communication system might be expected to re- 
ceive a large fraction of its needed input from a comprehensive evaluation 
system. Therefore, the information generated by the Program Approval and 
Evaluation Unit of the Division should be expected to provide a substantial 
amount of all information needed by the Division and its external audiences. 
It should closely link local education agencies with the Division and should, 
in addition to the information needed for use just within the Division, con- 
tribute substantially to the contents of reports for other audiences. 
O 13 
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Sources of Division Goals and Objectives 

Externally Initiated Objectives 

The Division is, by its control and function, an agency that most often 
responds to goals and objectives set elsewhere rather than establishing its 
own goals. The Division must, if it is to receive Federal funds for disburse 
ment, respond to Federal laws with specified plans and to plans submitted by 
local educational agencies. The Division must also respond to State of Illi- 
nois requirements for the funds its legislature provides, to the advice of 
formally organized advisory councils and to innumerable other documented, 
freshly written and spoken concerns for guidelines from educational agencies, 
groups, and individuals. 

Some understanding of the responsive position of the Division can be 
acquired by casual reading of a recent document issued by the Office of the 
Superintendent of Public Instruction. Action Goals for the Seventies was 
printed in May 1972. It presents nine substantive goals for education in 
Illinois and dozens of "action objectives" which arose through the convening 
of state-wide regional conferences a year earlier. The author of a draft 
document stated: 

The purpose of the Statewide Conference on Educational Goals 
and Priorities will be to draft a series of priority goal 
statements based upon the inputs of the six regional public 
hearings . 1 



^Ison, Thomas, A Proposal for Strategic Planning and Priority Goal 
Establishment for the Office of the Superintendent of Public Instruction 
(Draft copy), June 3, 1971. 
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Among the objectives that are listed in Action Goals for the Seventies 2 



are several which pertain directly to the Board of Vocational Education and 
to the Division: 

Statewide Governance 
Action Objective #1: 

By 1972, the Board of Vocational Education will be Placed 
Directly under the Office of the Superintendent of Public 
Instruction (p. 84). 

Career Education 
Action Objective #1: 

By July 1973, a System of Financial Support for Career 
Education will be Developed Consistent with Other Edu- 
cational Funding Practices in the State (p. 102). » 

Action Objective #8: 

By 1976 t a System for Training Career Education Profes- 
sionals will Provide Sufficient Personnel to Meet the Ca- 
reer Education Objectives (p. HO). 

Professional Qualifications and Certification 
Action Objective #1: 

By 1974, Certification will be Based on an Assessment of 
Satisfactory Classroom Performance Standards in Addi- 
tion to Credit-Hour Requirements (p. 110). 

Action Objective #4: 

By 197* , in Cooperation with the Division of Vocational and 
Technical Education, Aid in the Development of Grades K-6 
Career Awareness Programs, Grades 7-10 Career Exploration 
Programs, and Grades 10-12 Career Orientation Programs (p. 115). 



2 Action Goals for the Sev enties. An Agenda for Illinoi s E ducation . 
State of Illinois: Office of the Superintendent of Public Instruction. 
May 1972. 
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Inclusion of the Board of Vocational Education within the Office of State 
Superintendent of Public Instruction would have profound implications for 
planning, organization, and the conduct of Division programs. Career Edu- 
cation programs constitute a large part of the operations of the Division. 
Requirements for statewide assessment of either teacher or student attain- 
ments, which are matters of both national and state concern at this time, 
could dramatically change needs for evaluation performance in the Division. 
Indeed, it is possible that student performance assessment would be easier 
to initiate for many program outcomes sought by the Division than for reg- 
ular academic programs such as reading for which assessment is most often 
proposed. 

The objectives listed in Action Goals for the Seventies have not, of 
course, become part of a statement of Division objectives. But other sets 
of objectives (and statements of purposes, goals, tasks, and benefits which 
are similar in content to objectives) are reflected in Division documents • 
The Vocational Education Amendments of 1968 > which required the creation 
of national and state advisory Councils with prescribed duties and member- 
ship requirements, specified that a state plan for Vocational Education be 
prepared, and gave many requirements for its contents. The statutes of 
this law, while not stated with use of the term "objective" can be trans- 
lated into statements of objectives for division and local education agencies. 

Another source of external involvement in the statement of Division- 
concerned objectives was sponsored by the Division itself. Its Program Ap- 
proval and Evaluation Unit selected, through contracts with individuals 
throughout the state who had contributed to the preparation of a bank of ob- 
jectives, first and second priority objectives for referenced use by t\\e 
Division. Thus these objectives, which reflect judgments and wishes of 



ERLC 



16 



external parties, have become incorporated as objectives for operational 

educational agencies and for the Division itself* 

A recent addition to external influences on the formulation and use 
of objectives within the Division has arisen from the Vocational Education 
Amendments of 1968. The Advisory Council on Vocational Education for the 
State of Illinois has urged, through the submission of recommendations to 
the State Board, expansion of programs, improved surveillance of programs, 
services and activities, the addition of a requirement that schools provide 
for a system of placement follovup, and support for adult programs* 

Internally Initiated Objectives 

A transition from external influences to internal influences on the 
statement of objectives can be discerned in a document nominally distrib- 
uted "for discussion purposes only — Working Draft* 11 The document is titled 
A Systems Approach for Management of Vocational Education ; it was re- 
vised September 1, 1971* The document provides the best available summary 
of management information system thinking for the Division* It uses as its 
general purpose statement of the Division "suprasystem" the following state- 
ments from Public Law 90-576, Title I, Part A: 

It is the purpose of the Division of Vocational and Technical 
Education to maintain, extend, and improve existing programs 
of Vocational Education and to develop new programs of Voca- 
tional Education, which are realistic in the light of actual 
or anticipated opportunities for gainful employments and which 
are suited to the needs, interests, and ability of chose bene- 
fiting from such training. 3 



3 A Systems Approach for Management of Vocational Education * For 
Discussion Purposes Only — Working Draft, Revised September 1» 1971 * 
State of Illinois, Board of Vocational Education & Rehabilitation* 
Division of Vocational and Technical Education* 
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Since divisions of vocational and technical education in all states mxst 
report tc a common agency — the Federal Government , it should not be surprising 
that objectives selected for statewide use within Illinois would have much in 
coranon with objectives formulated in other states. The draft version of the 
Illinois State Plan for Fiscal Year 1973 1 * and that for the preceding year each 
contained statements of objectives prepared or selected and then assembled in- 
ternally. The proposed Fiscal Year 1973 Plan contains a new section (Part II 
Annual and Long-Range Program Plan Provisions). In this section are brought 
together, in a format different from that used in 1972, numerous objectives 
used previously in New York and possibly other states together with new object- 
tives prepared by the Division. The objectives in these documents include 
both statements calling for specific services to be provided by the Division 
(i.e. the provision of planning services to local education agencies) and 
statements calling for increased enrollments and student achievements. 

The management information system document that was referred to pre- 
viously contains a number of additional objectives, but these are objectives 
of the Division itself. That is, they call for the Division to plan, docu- 
ment, visit, maintain, and establish programs. These objectives, when taken 
in combination with those already referred to as having arisen either outside 
or within the Division, accumulate to produce a pool of many hundreds of ob- 
jectives which can be considered as one tries to learn what the Division is 
attempting to accomplish. The number of objectives is too overwhelming to 



Proposed State Plan for the Administration of Vocational and Technical 
Education in Illinois FY 1973 . Final Draft March 30, 1972. State of Illinois 
Board of Vocational Education and Rehabilitation, Division of Vocational and 
Technical Education. 
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comprehend without summary. Some are very easy to understand; others are 
only partially understandable; and some use words In their formulation 
which make them almost unintelligible. It was from this pool of objectives 
and statements that information requirements for evaluation were &«sembled. 

Summary Information Requirements 

Definition of Terms 

It is common and popular today to talk of information systems in 
education. The words used in discussions of this type and the words used 
to describe what educational programs are attempting to accomplish consist- 
ently do not have clear common meaning* The same words, for example 
"goal" and "objective", may have precisely interchanged meanings in two 
different educational agencies. To avoid imposing unnecessary terminolog- 
ical distinctions, the definitions of "purpose", "goal", "objective", and 
"activity" used in this document are, as much as possible, the same as 
those used by the Division in Ite preparation of the Management document. 
They follow: 

PURPOSES 

Purposes are not directed toward a specific time frame. Purposes 
represent a broad subjective review which is in turn utilized as 
a base upcn which the total system is built. Purposes are 
arranged in a hierarchical structure displaying the broad highest 
level purposes and the more specific lower-level purposes that 
support them. 

GOALS 

Each unit (system) must forecast areas of endetvor within a spe- 
cific time frame. The term "Goal" is used to define these areas 
of activity. By definition, a "Goal" is a written statement which 
indicates the duration and establishes the scope of a given number 
of related processes; It serves as a general statement to identify 
the nature of the activities to be performed. 
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OBJECTIVES 



In contrast to the goal, an objective is a statement of specific 
behavior. An objective is a written statement describing how the 
goals will.be accomplished and specified priority areas to be 
achieved in Fiscal Year 1972. 



ACTIVITIES 

Objectives are utilized to define the specific behavior necessary 
for accomplishing specific goals. At this point, activities must 
be defined to achieve the desired terminal behavior as stated in 
the objective. During this phase, management of physical resources 
requires being cognizant of program budgeting procedures as they 
related to purposes, objectives, and programs. 5 



This decision does, however, immediately produce a necessity for recog- 
nizing that words used to state "goals," "objectives," "activities," and 
"benefits" in the proposed State Plan for Fiscal Year 1973 were not formulated 
with these distinctions in mind. Indeed, statements in the Plan and in its 
predecessor document intermingle usages. For example, a goal to "encourage 
youth to be responsive to the welfare of others (p. 108, FY 73) and an objec- 
tive to "support studies designed to provide local and state planners with 

valid information " (p. 136, FY 73) are very similar. The intermingling 

of usage, while not unusual in complex planning documents, does make necessary 
constant recognition that the standardization of terms and meanings is a task 
that cannot be f >ly accomplished; tolerance for amtiguity in terms must be 
maintained . 



5 A Systems Approach for Management of Vocational Education. For 
Discussion Purposes Only—Working Draft, Revised September 1, 1971. 
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Information Needs 

It became evident from the number of objectives listed in the 
documents reviewed that no feasible way existed to enter each individu- 
ally into a statement of information needs. For the sake of clarity, a 
way of combining objectives was required. The method that has been 
adopted called for information needs statements to be derived from state- 
m# ts of goals, objectives, and so forth, present in the documents. In- 
formation needs statements were, furthermore, sought that would describe 
vocational education, outcomes external to the Division (that is, within 
and about local education agencies) and those which might be needed to 
describe internal Division functioning. 

Educational Testing Service staff members, after reviewing the 
materials available to them, sought to prepare a summary set of objectively 
stated information needs statements against which the performance of the 
Three Phase System could be compared. The preparation of this list consti- 
tuted a response to the first of four investigative questions: 

1. What information is needed? 

2. Does the Three Phase System seek to give 
attention to the need? 

3. How well is the System performing in 
response to the need? 

4. What recommendations can be made to improve 
System performance? 

The Three Phase System seeks to attend to all but perhaps the first of the 

listed summary needs — the one having concern for student skill attainment. 

Thus it appeared reasonable to attempt to focus on all fourteen information 

needs statements as guides for tracing the effectiveness of the Three Phase 

System. 
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The seven summary needs statements for information about the perfor- 
mance of individuals and agencies external to DVTE were: 

1* What measurable job entry skills do students possess 
when they complete programs? 

2. Are the numbers of agencies, programs and support 
experiences available to students increasing? 

3. Are the numbers and qualifications of available 
teachers increasing?. 

A. Are educational agencies acquiring adequate infor- 
mation by which to plan improved instructional offerings? 

5* Are guidance personnel and services being improved? 

6* Are program enrollments increasing and student retention 
being improved? 

7* Are students being assisted with placement in employment 
and followed up when they leave Immediate control of their 
educational setting? 

The six summary needs statements for information that are held by 

the Division were: 

1* What priorities should be set for program development and 
the provision of services? 

2. Are funds being correctly spent? 

3* Is the evaluation System reactant; that is, does it 
possess features for self-correction? 

A, Are agencies receiving consultation and support as required? 

5. Are business, industry and labor being effectively involved 
in program development? 

6. Do supplies of vocational education graduates meet the 
changing manpower demands in Illinois? 

These questions and answers to them are later summarized in Table 5 on 
pages 56 and 57* 
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METHODOLOGY 

ETS's analysis of the performance of the Three Phase System was initiated 
by careful study of the many documents prepared by DVTE, interviewing of per- 
sons engaged in the administration of the System, and study of records main- 
tained by the Program Approval and Evaluation Unit at its Springfield offices 
and at the University of Illinois - dhampaign where data processing and much 
organizational work was conducted. The elaborate nature of the System with 
its many forms, documents, and internal evaluation processes was generating 
far more information than could conceivably be exhaustively studied in the 
period of time available. Therefore, a sampling frame was first established 
to select limited quantities of data that could be examined carefully. Sub- 
sequent to preparation of the sampling frame, questionnaires were designed. 
A plan for the analysis of evaluation report data was structured, and supple- 
mentary studies were planned. 

Sampling of Educational Agencies 
Full scale implementation. of the Three Phase System during Fiscal Year 
1972 included the conduct of site visitations to a selected sample of the 600- 
plus Illinois agencies that submitted plans. Seventy-one agencies were visi- 
ted during the year. All of these agencies, whether full data was available 
for them or not, were included in the basic sampling frame. At the time sam- 
pling was conducted, 113 agencies were tentatively scheduled for visitations 
during Fiscal Year 1973. A restricted random sample of 35 of the 113, with 
the restriction that five be located in each of the six regions and from Cook 
County was drawn. A second group of 35 agencies was selected in the same man- 
ner from a summary list of agencies that had submitted local plans for Fiscal 
Year 1973 but had not been visited during Fiscal Year 1972. This list was 
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additionally adjusted to assure that proportions of postsecondary institu- 
tions would be approximately balanced across the three groups. 

The resultant combined list constituted the study population for this 
project. Because full data was usually available for 70 of the 71 agencies 
visited during Fiscal Year 1972, the numbers of agencies visited during Fis- 
cal Year 1972 were equalled by the number of agencies sampled from the other 
two lists. The combined population of 140 agencies is not perfectly repre- 
sentative of either agencies submitting plai s in 1972 or for Fiscal Year 1973, 
The lack of representiveness was due in part to lack of a final listing from 
which to sample but also because agencies did not have equal probabilities 
of inclusion in the sample. However, as will be indicated later, the agencies 
that were visited in Fiscal Year 1972 were not themselves a random sample of 
all agencies submitting plans for that year. Therefore, it was not possible 
to attain a fully representative sample of agencies. 

Questionnaire to Preparers of Plans 

After Division and Unit personnel had been interviewed, and available 
documents studied, a questionnaire (see Appendix) was prepared for mailing 
to persons who were involved in the writing of plans on behalf of LEA 1 s. 
This questionnaire, which was accompanied by a covering letter from the Ex- 
ecutive Director of the Advisory Council, was distributed to 137 individuals 
whose names were submitted to the Division as planners who signed and were 
known to have been involved in program planning. (Some planners signed but 
were not actively involved in planning. It is, for^ex^ple, customary for 
the superintendent of the Chicago Public Schools to sign such documents even 
though he devotes no time to their actual preparation; therefore, he did not 
receive a questionnaire*) 
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The questionnaire contained 27 items, one of which had 10 subitems. It 
was accompanied by an envelope, self-addressed and stamped for return to ETS. 
After five weeks, 114 of the recipients (83 percent) returned their question- 
naire in time for processing and inclusion in this report. 

Questionnaire to Visiting Team Members 
A second questionnaire (See Appendix) was prepared for mailing to mem- 
bers of the visiting teams. This questionnaire was not distributed to all 
team members; rather, while all persons were included whose descriptions sug- 
gested they fcare not professional educators, it was decided that only a sam- 
ple of the educators was needed. The number of noneducators to whom the ques- 
tionnaire was mailed totaled 88; the number mailed to educators totaled 60. 
The questionnaire for team members contained 10 items, one of which had eight 
subcategories and another had seven. These questionnaires were also accom- 
panied by an envelope addressed to ETS; after five weeks 61 (69 percent) ques- 
tionnaires were returned by noneducators and 57 (95 percent) were returned by 
educators. 

Analysis of Evaluation Report Data 
As was mentioned earlier, site visitations are followed by the printing 
and formal submission of an "Evaluation Report" of the occupational programs 
of the agency. These composite reports included profiles of the type shown 
in Figure C on the following page. 1 



* An Overview of the Three Phase System for Statewide Evaluation of 
Occupational Education Programs * Springfield: State of Illinois: Board 
of Vocational Education and Rehabilitation. 
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AREAS OF CONCERN 



RATING 



Poor 



Excellent 



•1. Administrative Oryanization 



3. Objectives 



2. Personnel 




4. Evaluation 




5- Occupational Programs 




6- Resources Utilized 




Z Guidance Services 




8. Students Served 




Figure C 
Example of Bar Graph in Evaluation Report 



The .profiles summarize for readers of the report the impressions of the 
visiting team. It is not possible to determine the validity of the ratings 
by analysis of the content, but analysis of them can contribute answers to 
questions like the following: Do personnel always receive high ratings? 
Does everyone get the same rating? So some teams give biasedly higher or 
lower ratings than other teams? Therefore, the analysis of ratings received 
by the agencies that were visited in 1971-72 was part of this project. Nur- 
merical values were assigned to each of the ratings in the eight areas of 
concern whereupon a variety of analyses were conducted for them. 



One of the goals of the System is to "help assure accountability of 
Federal and State funds allocated to local occupational programs." 2 With 

2 Three Phase System for Statewide Evaluation of Occupational Education 
Programs , op . cit . , p. 2. 



Biscal Audit Outcome Analysis 
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the collaboration of the Fiscal and Statistical Unit of the Division, arrange- 
ments were made to have Fiscal Year 1969 financial audits conducted prior to 
team visits* It was thereby possible foe teams to receive as part of their 
orientation materials results of the fiscal audit. While it is evident that 
the audit would be conducted for time periods that occurred prior to initia- 
tion of the Three Phase System (or even of the employment of its current 
staff), this effort to coordinate site visitation with fiscal accountability 
concerns was judged to be an important effort worth special study. Conse- 
quently, records of the Fiscal and Statistical Unit that included data for 
the agencies in our basic sample were examined; data was extracted from them 
for summary and reporting. 

Study of Trends in Statewide Evaluation 
An external (to the Division and its functions) survey project undertook 
to determine national trends in statewide evaluation which might be considered 
important for the Division., These trends might be important both as stan- 
dards against which Division performance might be judged and for the ideas, 
they would produce that could guide and benefit the Division in its efforts 
to improve evaluation practices. Dr. Gordon Law, Professor, Department of 
Vocational-Technical Education at Rutgers University, who is a nationally 
recognized scholar and consultant to state agencies studying their evaluation 
systems, in collaboration with Dr. Raymond Wasdyke of ETS at Princeton, under- 
took a literature search and produced a critical summary from it. The result 
of that investigation is, with modifications, incorporated as a section later 
in this document. 

Miscellaneous Studies 
Not all of the many lesser studies that were conducted i.eed special 
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identification and notation. However, two additional studies merit mention. 
One was a study of the pattern and trends of submissions of claims for pay- 
ment for elementary school (K-8) programs. This study was initiated because 
national statistics for the impact of this funding had been heavily influ- 
enced in prior years by a very large increase in pupils enrolled in the state. 
The study that was undertaken extracted for the basic sample of schools the 
actual and estimated enrollments for the program as recorded in Section G - 
Summary of Local District Plan. 

A second study called for the identification of new program initiation 
in sampled agencies during 1971-72 and of proposals for new program initia- 
tion by the same agencies for 1972-73. Data for this study was accumulated 
from two sources. Mid-year reimbursement records generated by the Fiscal 
and Statistical Unit were studied to acquire identifications of initial 
programs funding during 1971-72; new program submissions for Fiscal Year 
1972-73 were copied from Section D entries of the Annual Plan submitted for 
that year. Entries from these records were coded on special data sheets to 
facilitate keypunching for later analysis. 
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OUTCOMES 



Analysis of the Three Phase System 
Phase I: Development of Local Plans 

In 1970 the Research and Development Unit of the Division funded Research 
Project RDC-A1-029 entitled Study of Local Plans Submitted bv School Districts 
in Illinois to the Board of Vocational Education and Rehabilitation. The study 
was directed by Dr. William Rogge of the University of Illinois. A final report 
draft was submitted in June 1971. 1 This study, which involved many month3 of 
plan reading and interviewing of local planners for high school districts, re- 
sulted in an extremely useful document. Prior to the conduct of visits to 87 
high school districts, a total of 293 local plans subntitted to the Board of 
Vocational Education and Rehabilitation were reviewed. 

Because the work conducted by Dr. Rogge 's group was extremely complete, 
the ETS study of Phase I activities did not attempt to repeat it. Instead, 
this investigation attempted to update and extend knowledge that had already 
been acquired. A large fraction of the items that were included in the ques- 
tionnaire for program planners was linked to the Rogge report. The actual 
reading of plans for this study was undertaken with the same questions in mind. 
In the sections that follow, the Rogge report and then our supplemental research 
are discussed. 

Rogge 1 s Local Plan Study 

The Study of Local Plans was designed to answer four questions. Quota- 
tions of those questions with some elaborative information eliminated follow: 



Hfilliam M. Rogge, Study of Local Plars Submitted bv School Districts in 
Illinois to the Board of Vocational Education and Rehabilitation Urbana- 
Champaign: University of Illinois, June 1971. 
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1. To what extent does the local district use a planning process? 

2. To what extent does local planning incorporate the high 
priorities of the 1968 Vocational Amendments and the State 
of Illinois Guidelines? 

3. To what extent do local plans (written documents) reflect the 
actual intentions and activities of the school district? 

4. To what extent are local plans actually used as a document in 
restructuring local programs, especially in the development 
of new programs av:d in the inservice training of staff? 2 

The analysis which was reported in the Final Report Draft heavily 
emphasized concern for change process in the utility of agency planning 
as a device for producing improved educational programs. This approach 
is consistent with a generally unstated purpose of the Three Phase System, 
which is to stimulate educational change within applying institutions. • The 
report maintains consistent concern for the change-inducing potential of 
the Phase I planning process throughout its narrative. At the end of the 
document an extensive list of conclusions was drawn. A partially edited 
list of conclusions follows. 

1. The requirement to prepare a local plan for the state 
so far has resulted primarily in giving those specifics that 
will assure reimbursement. Other sections tend to remain gen- 
eral and are not taken very seriously in follow-through activities. 

2. Partly or largely because of reimbursement being tied 
to course offerings, a majority of the changes perceived as sig- 
nificant at the local level are course additions or changes. 

3. Local planning in each school district fails to meet many 
or most of the criteria for good planning. Even those criteria 
most emphasized, such as collecting and using data, are not met 

in most school districts. 

a. Planners are almost completely vocational teachers, 
administrators, and counselors. Almost no students and 
few laymen are involved. 

b. Planning time is limited; few plarring resources are 
available; outside consultants are primarily "content" 
experts . 



2 



Ibid ., p. 2. 
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c. Background data, such as job opportunity surveys are 
often available in some areas but not others; available 
data often is not used seriously. 

d. National priorities are examined more for reimubrsement 
funds tied to them than for the underlying reasons they were 
established. 

e. Creative problem solving is almost non-existent in local 
planning. 

f . The internal conmunication of the planners is not singled 
out as an aspect of planning to be improved. 

g. The goals set for local vocational programs are too 
broad and general to be used for focused planning. 

h. The recent developments for "systems" planning are almost 
completely unused. 

i. Follow-through that concentrates on changing the behaviors 
of people is limited to verbal, cognitive, symbolic behaviors, 
and not those behaviors evident in such procedures as micro- 
teaching. 

j. Evaluation of the planning process is almost unknown. 

4. The expectation that local planning shall be done by local 
school districts is, in itself, not a sufficient force to produce 
effective planning. Other inputs are required » including: 

a. Financial support specifically earmarked for planning. 

b. Training of local personnel to lead planning activities. 

c. Outside consultants trained to help in the process of 
planning. 

d. Reimbursements also tied to the use of evaluation aad 
follow-through • 

5. Vocational planning at the local level is dominated by one 
or a few persons, mostly teachers, counselors, and administrators. 

6. Most local districts are not large enough to support com- 
prehensive planning activities. Too many other immediate concerns 
rate a higher priority. 

7. Other than the addition of courses, the many high priorities 
of federal legislation and state guidelines are not being met through 
existing local planning at a level that probably will be acceptable 

to legislators. 

a. Programs for the disadvantaged are mostly at the stage of 
identification and stating objectives, but are not comprehensively 
carried out. Some notable exceptions exist, however. 

b. Pre-vocational programs are not seriously being planned in 
most school districts. 

c. Handicapped children are perceived by vocational educators 
as being provided for primarily by existing special education 
programs. 

d. Counseling and guidance services are being expanded but 
few new approaches are being developed. 

e. New occupational areas are being developed too slowly 
probably to satisfy congressional demands. The changes are 
too limited to revamping existing courses and programs. 
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The Rogge report concludes with the presentation of four major recom- 
mendations that support (1) the use of state reimbursement funds for plan- 
ning, inservice training and evaluation 9 (2) regional planning to assist 9 
in particular, small school districts, (3) use of training programs to im- 
prove planning, and (4) service by educators in internships in industry and 
government, and by noneducators in vocational education. 

Outcomes of ETS Questionnaire Study 
Many of the items that were included in the Questionnaire for Program 
Planners were used to learn how respondents viewed the planning process in 
September 1972. Two items (Items 10 and 17) asked about the help being given 
by the staff of the Planning and Evaluation Unit of the DVTE to planners. A 
strong majority of respondents seemed to feel that the Unit was attempting to 
be helpful and was successful in its effort. Whatever the need for external 
assistance was, it would seem partly necessitated by lack of clarity in in- 
structions received on forms of the Division. Forty percent of respondents 
disagreed with Item 1 which stated that instructions were "clear and easily 
understood ;" 43 percent responded in the same manner to Item 22 which stated 
that guidelines M helped us to develop needed new programs." Similarly, a ma- 
jority of respondents to Item 5 disagreed that the Division was dc?ng a good 
job of helping them "to understand how to meet financial accountability 
requirements." 1 * 

Responses to Item 9 suggest that the planning effort did not become eas- 
ier with time. That is, 42 percent of respondents to the item indicated that 

3 Ibid. . p. 31. 

**The percent of response to each, item of the questionnaire is presented 
on the sample instrument included in the Appendix. 
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it took as much or more time to prepatv the 1972-73 plan as it did in 1971-72. 
One problem in planning was to know T *ov to identify or classify handicapped 
students. Responses to Item 11 suggest that about 3 out of 4 planners be- 
lieved they had trouble knowing which students should be classified as handi- 
capped. 

Good planning should, authorities say, involve the use of Advisory Com- 
mittees, students, and other individuals with special understanding. Several 
items were included to learn about the use of these other persons. Responses 
to Item A suggest that employer representatives were helpful contributors in 
planning. A small majority of respondents to Item 13 indicated that students 
were "a big help" to them in their curriculum development. A majority of re- 
spondents to Item 18 expressed confidence that employers knew enough about 
skills their employees need to be helpful in program development. However, 
responses to Item 21 indicated that i was not easy to involve Advisory Com- 
mittee ©embers in planning. 

Ij availability of fully modem equipment important to occupational pro- 
grams? This issue *?as submitted as Item 19* About one-third of those who 
answered the item indicated they believed that having fully modern equipment 
was "not important 11 for occupational programs; the remainder disagreed with 
this position. 

The planning process can itself be very expensive. Item 12 was included 
to determine whether respondents believed the cost of planning could be justi- 
fied by the amount of money received. Respondents were about equally divided 
on this item indicating that the -oount of money received was not sufficiently 
high to convince most planners that the effort was clearly justified by the 
resultant reimbursement. 

One block of items (those for Item 6) was included to seek current veri- 
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fixation that recommendations of the R^gge Report for the need for external 
assistance in planning still existed. These items had concern for each of 
many types of assistance. Responses to them seem clearly to indicate that 
assistance was thought to be needed for (1) the conduct of stu- nt followup 
studies, (2) the conduct of local community needs surveys, (3) the conduct 
of employer needs surveys, (4) ( the preparation of behavioral objectives, 
(5) the conduct of evaluation, (6) the identification of job entry skills 
required of students, (7) the setting of priorities for program development, 
and (8) the actual preparation of comprehensive planning documents. Two 
types of assistance were less important in the minds of respondents. 
Slightly more than half disagreed that help was needed to conduct student 
interest surveys; because of help already provided to them by the Division 
or for other reasons, only one-fourth felt they needed help in the organi- 
zation of Advisory Committees. 

Responses to the items of this questionnaire strongly support the con- 
clusions drawn in the Rogge Report. They indicate that the personnel of 
the Division were generally rated as helpful, that some of the printed mate- 
rials were difficult to use, and that a substantial amount of external assist- 
ance was needed. From this it can be seen that the planning documents pro- 
vided by the Division at the time this study wa3 conducted were regarded as 
falling short of planner's needs; improvement of the documents should be 
sought. 

Phase II Approval of Local Plans 

Various kinds of information were collected in an effort to throw addi- 
tional light on the influence of evaluation procedures on the rate of accept- 
ance of plans reviewed by the Department of Vocational and Technical Educa- 
tion. In an attempt to get a f <s type of information, the Status of Plan 
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Reports were checked for the 141 institutions included in our sample. The 
following tables shows the results for the three groups of institutions: 
first, those schools that were visited daring the 1971-72 academic year; 
second, those that were not visited in 1971-72 but will be isited in 1972- 
73; and third, those schools which were not visited in 1971-72 nor are there 
plans to visit in 1972-73. 



Table 1 

APPROVAL STATUS OF PLANS AS OF SEPTEMBER 8, 1972 



Program 
Year 



71- 72 

72- 73 



A* 



CA 



NA 



NA 

to 
CA 



NA 
to 

A Total 



Agencies Visited 1971-72 



46 
40 



25 
16 



0 
7 



0 
3 



0 
5 



71 
71 



Totals Initially Not Approved (NA) 
Agencies Visited 1971-72 



0 
15 



21% of 71 



71- 72 

72- 73 



71- 72 

72- 73 



Agencies to be Visited in 1972-73 



15 
17 



19 
18 



1 
0 



0 
0 



0 
0 



35 
35 



Agencies not Scheduled for Visits 



11 
15 



22 
18 



0 
1 



2 
1 



0 
0 



35 
35 



Agencies to be Visited in 1972-73 

1 3% of 35 
0 

Agencies not Scheduled for Visits 



2 
2 



6% of 35 
6% of 35 



71- 72 

72- 73 



Totals 

72 
72 



66 
52 



1 
8 



2 
4 



0 
5 



141 
141 



Totals 



3 21% of 141 
17 12% of 141 



*A Fully Approved 

CA Conditionally Approved 

NA Not Approved 
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Of the 71 schools visited in 1971-72 , 46 were fully approved (A) on the 
first round of readings and 25 were conditionally approved (CA). For the same 
institutions in 1972-73, 40 were fully approved on the first round and 16 con- 
ditionally approved. A total of 15 were not approved which meant that 21 per- 
cent of the 71 institutions that were virited during the previous year were 
not initially approved* However, of the 15 schools, only 7 that were not ap- 
proved initially remained not approved (NA) at the time of this study. Five 
of the institutions that were not approved initially received full approval 
after submitting revised plans (NA changed to A) and three received conditional 
approval (NA changed to CA) . 

In contrast to this, for the group that will be visited in 1972-73 (a 
total of 35 institutions), 15 schools received approval after their initial 
reading in 1971-72 while 19 received conditional approval; only one of the 
35 institutions did not receive approval. For the same 35 institutions in 
1972-73, 17 received ~ approval and 18 received conditional approval. No 
school among the 35 to be visited in 1972-73 was disapproved initially this 
year. 

Substantially the same results were obtained for the 35 schools that were 
not scheduled for visits as for those that will be visited this year; 11 received 
full approval initially in 1971-72 while 22 received conditional approval. Two 
initially were not approved but after submission of revised plans received con- 
ditional approval. For 1972-73 these same 35 institutions were processed as 
follows: 15 were fully approved, 18 were conditionally approved, and 2 were 
not approved. Of the 2 that were not approved 1 received conditional approval 
after submitting a revised plan and the other had not received approval as this 
report was being prepared. 
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Summarizing the results across all 141 institutions, we find that for 
1971-72, 72 schools received full approval initially, 66 received conditional 
approval initially, and only 3 were not approved. Of the 3, 2 eventually re- 
ceived conditional approval of their plans. The following year, the distri- 
bution was as follows: 72 institutions received full approval initially, and 
52 received conditional approval. However, 17 (12 percent) of the schools 
were not approved initially. After the resubmission of the plans, 9 were ap- 
proved, 5 fully and 4 conditionally* As a result of the team visits and the 
evaluation reports which teams prepared to describe the programs in the 71 
institutions, there was a sharp increase in the number of plans that were 
not initially approved. This percentage went from 2.1 percent in 1971-72 to 
12 percent in 1972-73, indicating that site visitation contributed information 
used to disapprove programs. Without this information there had earlier been 
few rejections. 

A further analysis of the 17 plans that were not first approved in 1972- 
73 reveals a difference in the rejection reasons given for those institutions 
that were visited and those that were not. Of the 15 schools that were vis- . 
ited the reasons generally related to failure of the schools to respond or 
comply to recommendations made by the evaluation team. For those that were 
not visited, the comments were very similar to those made by the Division the 
previous year. They primarily noted failure of the applying agency to state 
objectives and goals of the programs in behavioral terms or failure to meet 
one of the technical requirements of the Three Phase System. It seems evi- 
dent from an examination of reports that rhe kinds of deficiences found in 
vocational-technical programs by evaluation teams have had a marked influence 
on initial Status of Approval of plans by the Division after visitation has 
occurred. Summary Reports have been used to guide readers of nlans as they 
determine rejection or acceptance the following year. This could lead to 
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improvements in the quality of plans as well as the vocational and occupa- 
tional education programs being offered in the schools. 

The most common basis for rejection of 1972-73 proposals seems to have 
arisen because agencies did not specifically deal with recommendations of 
the visitation team when submitting their continuation proposal. However, 
specific references were also often made to lack of behaviorally stated ob- 
jectives; other reasons, none with high frequency, included need to appoint 
a Director of Vocational Education, the necessity of organizing an Advisory 
Council or take steps to do so, or to attempt to improve counseling. 

Whether the reasons for nonapproval would lead, when responded to, to 
improved programs for students can only be surmised. Ft*' example, -it is not 
at all clear how stating objectives behaviorally would lead to important im- 
provements in program outcomes. Similarly, the general exercise of'respond- 
ing in writing to recommendations can conceivably contribute very little to 
program alteration. 

Records of the Program Approval and Evaluation Unit also indicate that a 
heavy amount of follow-up effort was directed toward helping agencies that 
had been visited. Memoranda and notes attest to very extensive transfer of 
information about agency needs to other units of the Division for action. 
It must be accepted that follow-up was not left to chance. 

Two items in the questionnaire for planners requested information 
about the review process, item 2 requested judgments of fairness of the 
proposal review process. Less than 20 percent of respondents disagreed that 
proposals would be reviewed fairly and on schedule. However, about one- 
third of all respondents indicated disagreement that they understood the 
criteria by which plans were to be reviewed and judged. This latter 
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response suggests that some plan writers, for whatever reason, had seri- 
ous difficulty anticipating how criteria might be applied to their plans; 
this lack of understanding can become more apparent and evident as the 
visitation phase of the System is used for additional agencies. 5 

Phase III; On-Site Visitations 

Visitation teams consisted of from three to more than 20 individuals, one 
or two of whom were designated to be team leaders. No single formula existed 
or could be established whereby the vocational characteristics of team members 
could be precisely stated. However, each of the different types of persons 
who were to be sought and invited to be team members could be found in the list 
for one or more Agencies. 

In this section the distances between the sites to be visited and the 
work or home addresses of visitors is the first point of discussion after which 
outcomes of the visitation and its follow-up are attended to. 

Distances Team Leaders and Members Lived from Agencies Visited 

It is often suggested that evaluative site visitations suffer heavily 
from the development of excessively friendly relationships between the vis- 
itors and personnel of the sites they visit. If reciprocal visitations occur 
over years it is suggested that reciprocity may also appear in ratings that 
are given, with the result that ratings become very similar and can be ex- 
pected to be consistently favorable. A counter argument used to justify re- 
cruitment of visitors from nearby locations is that this reduces travel costs 
through reduction of out-of-pocket expenditures as well as by the elimination 
of travel time. 

5 Approval rates for plans submitted by local agencies in Fiscal Years 
1970-72 were summarized on p. 34 of the Annual Descriptive Report for July 1, 
19?0-June 30, 1971 (Bulletin No. 33-372). 
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To determine from what distance team leaders and team members were re- 
cruited, a tally was made of estimated linear distances, in miles, between 
home or work address of the team member and the location where a visit was 
conducted. These distances, which are less than highway mileages, were 
read from a standard mileage map for Illinois. Error of necessity exists 
in this measurement process but its amount should not exceed an average of 
two or three miles. The median mileage estimate for the 69 sites that were 
included in this tabulation was 38; the range of linear mileages is from 
12 to 114 with the combined distribution shown in Table 2. 

Table 2 

Linear Miles Between Home or Work Locations of 
Team Members and the Agencies they Visited 





Number 


of Agencies 


Range of Miles 


In Interval. 


Cumulative 


10-19 


4 


4 


20-29 


15 


19 


30-39 


19 


38 


40-49 


16 


54 


50-59 


8 


62 


60-69 


3 


65 


70-79 






80-89 


3 


68 


90-99 






100-109 






110-119 


1 


69 



It can be seen from this distribution that most team members were re- 
cruited from less than fifty miles from the site they visited. The potential 
for developing close interpersonal relationships between visitors and visited 
and for indebtedness to each other for kindliness during the preparation of 
visitation reports cannot be discounted as a sour.ee of future concern. In 
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densely populated areas, fifty miles could serve as a very effective barrier 
between excessive personal familiarities; however, in rural areas, a distance 
of fifty miles might be accompanied by some degree of familiarity and sympathy 
for another's problems. 

Analysis of Evaluation Report Profile Data 

Each agency that was visited received an Evaluation Report. This re- 
port included both narrative and figural summaries. The eight bar profile 
constitutes the most visible and attention getting part of the report. It 
also provides, because of its quantitative nature, a basis for seeking in- 
formation about sources of bias that might have entered into the rating 
process. For this reason profile information for the agencies that were 
visited was coded on a scale from 0 to 5 for analysis. This data togethe 
with information identifying team leaders, sizes of teams, regions in 
which the agencies were located, types of agency and needs of agencies as 
reflected by ability (ABL) and adjustment (ADJ) reimbursement information 
were punched on data cards. This data was subsequently analyzed in a 
variety of ways with the results reported in the subsections that follow. 

Correlation Analyses 

Data from 68 of the 71 agencies that were visited was intensively an- 
alyzed. The mean and standard deviations of ratings on the eight profile 
levels are recorded in Table 3. 

A rating of 2.5 would be a mid-scale or average rating for the profile 
that is used. Of the eight levels, three received mean ratings below 2.5, 
although two of them were in the "average range 11 (2.0 to 3.0). The lowest 
ratings went to Evaluation which was the only rating falling in the "below 
average 11 range (1.0 to 2.0). The second lowest mean rating was given to 
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Statements of Objectives. The 2.4 average rating received by Guidance Ser- 
vices was only slightly below mid-scale. It is important to keep in mind, 
however, that these ratings reflect team perceptions of local potential for 
improvement. 

Highest ratings and ratings with the least variability associated with 
them were assigned to Personnel. This suggests that, as is quite common in 
professional personnel rating systems, visitors were unable or unwilling to 
make distinctions between different levels of personnel capability and there- 
fore issued relatively high ratings as an acceptable compromise to deal with 
these difficulties. It is interesting that only two of the eight ratings 
had means falling in the "above average" range (3.0 to 4.0); one went to 
Personnel and the other to Students Served. 

The category receiving the most variable ratings was Guidance Services. 
Table 3 shows that the standard deviation for this category was; 1.04. 

What correlations exist among these ratings, team size, and ability data? 
These are reported in Table 4. 

The inter-correlations among the eight composite ratings range from -.04 
to .62, with the largest number of them in the .4 f s. This suggests that there 
is similarity among the eight levels but not so much that they do not contain 
importantly different information from each other. 

Correlations between these ratings and both team size and ABL, however, 
tend to be negative. For example, a correlation of -.26 arose between size 
of the team and the Administrative Organization category, suggesting that 
those agencies having the largest number of programs were seen to be less 
effective in their efforts to arrange for their management than agencies hav- 
ing fewer programs. Guidance Services were also seen to be less effective in 
the larger agencies. 
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The correlation of -.38 that arose between ABL and the Students Served 
category indicates that more needy agencies in terms of financial ability to 
support programs from local funds received lower ratings on this category; 
indeed, the first eight negative correlations in the ABL column indicate 
consistency in tendencies of dams to assign lowest effectiveness ratings 
to agencies having greatest financial need. A potentially disturbing sta- 
tistic in the ABL column is the correlation of -.30 between (local) Re- 
sources Utilized and ABL which suggests agencies having greatest need tend- 
ed to make least effective use of resources available to them even though 
they have greatest need to do so; it is difficult to reconcile this 
relationship with consistently high composite ratings for personnel. Per- 
sonnel ratings may be made without appreciable concern for how the individ- 
ual staff member uses these resources. 

The negative relationships which appear between composite ratings and 

ABL also indicate that, as was reported in figures of the Composite Evalua- 
tion Report,* the ratings were, when recorded, still related "teams 
perception of local potential for achievement. 11 They imply that needier 
agencies received lower ratings even after local potential was taken into 
consideration* If that were the case it might be thought that needier agen- 
cies should receive even greater financial support than is now provided to 
them. However, because the System has, as yet, provided no firm information 
to describe how adjustments for local potential are made by team members, no 
useful meanings can be given to these negative relationship between System 
data elements; indeed, unless encouragement to make adjustments is either 
dropped or procedures for doing so are made clear the usefulness of compos- 
ite ratings may be difficult to defend over time. 

^ Composite Evaluation Report for Occupational Education in the State 
of Illinois Fiscal Year 1972 » Springfield: State of Illinois, Board of 
Vocational Education and Rehabilitation. pp. 7, 14, 18, 22, 28, 33, 37, 41. 
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Several summary conclusions can be made from the correlation analysis: 

1, Mean ratings foi- Evaluation and Statements of Objectives 
were lowest of the eight ratings, 

2, The mean rating for Personnel was highest and showed least 
variability. 

3, Ratings for Guidance services showed greatest variability, 

4, Negative correlations between composite ratings and team 
size indicate that larger agencies tended to receive lower 
ratings; this relationship was most pronounced for Admin- 
istrative Organization, 

5, Negative correlations between composite ratings and ABL 
indicate that neediest agencies tended to receive lower 
ratings, 

6, The stipulation that composite ratings include adjustment 
by team members for agency "local potential" importantly 
limits the conclusions that can be drawn from composite 
ratings • 

Variance Analyses 

A strong method for learning more about the effectiveness of Phase III 
activities involves analysis of the variability of ratings. The most im- 
portant of the analyses that were undertaken involved data lor 48 agencies 
and 16 different team leaders. The purpose of this analysis was to deter- 
mine whether team leaders, who often served with more than one team and for 
more than one agency, might contribute to unusually high or low ratings for 
the agencies they were assigned to. Sixteen team leaders had served with 
three different agencies. Therefore, an analysis was undertaken to deter- 
mine whether these leaders might have unknowingly or otherwise consistently 
contributed to lower or higher ratings for the agencies they visited. The 
analysis showed that mean ratings given to agencies differed in nonrandom 
ways across the team leaders. It also revealed that no evidence of system- 
atic interaction between the different leaders and the categories of the 
composite ratings could be identified. 
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Subsequent study of the type of agency visited by team leaders indi- 
cated that some had haZ greater responsibility for visits to agencies hav- 
ing loves' abilities as defined by the ability index (ABL). Conseqr^ntly, 
it must? be concluded that differences in composite ratings that were made un- 
der the direction of the different team leaders can be as easily attributed 
to agency features outside their control, as to the performance of the team 
leaders. No convincing evidence of the influence of bias by team leaders 
was found to exist. The same conclusion was appropriate for a supplemental 
analysis that included team leaders who had* visited but two agencies. 

Variance analyses were also performed to seek relationships among rat- 
ings assigned by team size, type of agency, and region of the state. Each 
of these analyses showec? no differences which would suggest the rating system 
as employed in the visitation phase differed in the various regions of the 
state, for different types of agencies, or team sizes. The negative correla- 
tions between agency size and ratings that were reported earlier were too 
small as a group to produce tin affect in team size variance analyses. 

The conclusion that can be drawn from analysis of the ratings is that 
evidence of bias associated with team leaders, size of teams, region of 
state, and type of agency cannot be isolated. However, biases could appear 
over time. It might be anticipated that biss could, if it were to appear, 
arise gradually as the system became more broadly used, and persons became 
more familiar with it. Recommendations whicli follow at a later point in 
this document are intended to contribute to the prevention of bias arising 
at a future time. 

Profile Rating Questionnaire Responses Analysis 

The questionnaire on profile ratings was designed for the purpose of 
learning more about how ratings were prepared a d how members of visitation 
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teams viewed the visitation process about a half year after participating 
in it. They were first asked how ratings were derived for the composite 
section. They responded that informal discussions were most often used, 
although quantitative ratings (presumably on a 0-5 scale) were often em- 
ployed. Both educator and noneducator respondents indicated that the pro- 
file ratings they would have assigned as an individual were similar to the 
responses that ultimately were assigned. 'This item was particularly import- 
ant because of the commonly held impression that noneducators might rate 
occupational programs lower than educators. This was clearly not the case. 
In fact, scrutiny of responses to all items in the questionnaire and par- 
ticularly those to Item D suggest that noneducators were, if anything, less 
critical in their ratings than educators. 

Responses to Item F revealed that team members believed their training 
to serve on the team was adequate (more than 80 percent of the respondents 
said "yes" to this item). More than 80 percent also said they were posi- 
tively impressed by the operation of the Three Phase Evaluation System 
(Item I) and expressed willingness to serve again as a team member if in- 
vited to do so (Item H) . 

Three out of four noneducators said they were positively impressed by what 
they learned about the programs of the agency; a similar number of educators 
agreed with this item. Both most often agreed, when responding to Item G, 
that visitation teams of five or more individuals should include at least two 
persons who are neither educators nor students. 

Item D requested a judgment of the efforts of agencies in each of seven 
areas. Efforts were judged to be most satisfactory for the acquisition of 
up-to-date equipment ard the development of work-study programs for which 
"yes" responses exceeded 66 percent. Efforts to develop sound working relation- 
ships with persons in the service area came next in ranking with "yes" responses 
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in the 60 percents. Noneducators (70 percent) and educators (53 percent) 
were less in agreement with regard to the efforts being made to determine 
skills needs of students. 

Lowest ratings were given to the efforts of agencies to place students 
in jobs, to determine the skills that program graduates possess, and to 
follow-up on students after they leave school. Ratings on the last three 
items were so low that they suggest increased efforts should be made in 
thos^ areas if programs are to be improved. 

Responses to Items in the Planners 9 Quesvionnaire 

Five items in the questionnaire for planners were to be responded to 
only by persons whose agencies had been visited in 1971-72. Responses to 
these items were generally very positive. For example, 44 of the 56 res- 
pondents agreed or strongly agreed that the visitation team prepared a fair 
and competent report (Item 23). Forty-five, in responding to Item 24, in- 
dicated they knew, at the conclusion of the visit, what had to be improved 
upon if the agencies 1 program plans for 1972-73 were to be approved . Forty- 
three of 56 who responded to Item 25 indicated team members were well pre- 
pared to do their work. 

In addition, 35 of 56 agreed or strongly agreed that they received good 
follow-up support from the Division after they were visited. This number 
seems low enough to wonder whether follov-up support was as strong as it 
might have beeiv^a question, the answer to which, is deferred to a later 
point . 

Forty-four of the 56 respondents to Item 27 indicated they believed 
the Three Phase System should be continued at least until each agency had 
been visited twice. 
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Six other items in the same questionnaire had concern for the visi- 
tation system and were responded to whether or not an agency had been vis- 
ited. Opinions were about evenly divided on the question of whether vis- 
iting team members should come from more than 50 miles from the agency they 
were to visit (Item 8). A strong majority (80 percent) indicated in re- 
sponses to Item 7 that schools to be visited in Fiscal Year 1973 would bene- 
fit from the preparation they would make for the visits. Less \an 20 per- 
cent expressed belief that site visitation would prove to be a waste of 
time and money (Item 20) . 

However, more than one-third of respondents suggested their counselors 
might not be well qualified to do occupational counseling (Item 14) although 
more than 72 percent believed the agency should formally help graduates and 
dropouts get placed in jobs (Item 15). Finally, in the responses to Item 16, 
74 percent said they would like to see their programs judged by the occupa- 
tional skills their students acquire. 

The data of this section suggests that the visitation system of the 
Three Phase System has been positively accepted and is being competently 
implemented. Recipients of ratings believed they were being fairly eval- 
uated by the System and that it was beneficial to them. They expressed 
these views even though their programs had not received rcompletely favor- 
able ratings. In fact, counselor qualifications, student follow-up and 
assessment of skills were often cited by them as weak spots in their pro- 
grams. Thus the rating system is providing information, not all of which 
is favorable, that agrees with the impressions of persons on site as planners. 
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Related Studies 

Fiscal Audit Analysis 

One goal of the Three Phase System is to help to assure accountability 
of federal and state funds allocated to local educational programs. 
Audits of agency records for previous years have been included in the design 
of the Three Phase System as a precedent: to site visitation. Therefore, 
prior to visitation in Fiscal Year 1972 many agencies received fiscal audits 
of their FY 1969 records. These audits could not reveal deficiencies in the 
Three Phase System, because the system did not exist at the time the over- or 
underpayments occurred; that is the System did not exist in Fiscal Year 1969. 
However, the problems which local education agencies had in understanding 
requirements for payment and maintaining records for payment are problems 
which might still exist and therefore were judged to be worthy of attention. 

The Control Sheet record of Fiscal Year 1972 funds that is maintained 
by the Fiscal and Statistical Unit of the Division contained entries made 
from January 4, 1972 to July 31, 1972 that refected a balance of $314,060.00, 
The entries that were made were largely for overpayments ($419,565.00 in debits) 
although underpayments were also recorded ($105,505.00 in credits). 

Agencies that received less than $5,000.00 in reimbursements were not 
audited. Auditors were primarily concerned with evidence of enrollments and 
were provided with guidelines for sampling of record books and other steps to 
determine what enrollments had been. Six codes were used to classify the ad- 
justments that were made: (1) Insufficient Record, (2) School Error, (3) Error 
in Springfield, (4) Not Substantiated, (5) Overclaimed Enrollment^-and (6) Over- 
claimed Units. Negotiations were sometimes conducted with agencies that did 
not agree with outcomes of the audit, whereupon amounts noted in original ad- 
justment letters were sometimes changed through use of an Adjustment Memorandum. 
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Not all of the agencies that were visited in 1971-72 received audits. 
Sampled agencies to be visited in 1972-73 showed debited adjustments in ex- 
cess of $10,000,00; sampled agencies not scheduled for visits showed debits 
of $246.00. This makes clear that agencies visited in 1971-72 tended to have 
larger programs than those not visited. Also, because exceptions were noted 
for each of the agencies that were audited, misunderstandings about require- 
ments for reimbursement were extremely widespread. 

Occupational Information Programs (K-8) Analysis 

The Section G K-8 enrollment statistics for local district plans for 
FY 73 were recorded as part of the data collection process. This data was 
collected primarily because of dramatic increases in Illinois enrollment 
during preceding years which had been noted by the Illinois Advisory Council. 7 
The increases had included enrollments for elementary occupational information. 
The data that was collected was intended to be used to identify continuation 
of trends that might be associated with approval processes. 

Thirty-three of the agencies that received site visitations in 1971-72 
reported K-8 data in their Section G's; 11 of the agencies to be visited in 
1972-73 and 16 of the agencies not yet scheduled for visitation also sub- 
mitted such data. Most, 42 of the 60 agencies in the combined group, making 
submission listed all students enrolled in the System as having benefits 
scheduled for them. This proportion represented an increase of more than 10 
percent over the preceding year with no important trends visible in the data. 
At the time this information was collected, the request for reimbursement that 
had been submitted for all students enrolled in the Chicago Public Schools 



Annual Evaluation Report FY 1971 . State of Illinois Advisory Council 
on Vocational Education, January 1972, p. 9. 
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,*> had not received approval. The critical decision to approve or eliminate 

C 

the reimbursement for large numbers of Chicago students (419,000) clearly 
f * results in a dramatic impact on enrollments for a fiscal year. 

New Program Initiation Analysis 

An effective evaluation system might be expected to have an impact 
upon both the numbers of new programs being produced each year and the im- 
portance of those programs as determined by priority levels assigned to 
them. Consequently, for each of the agencies visited in 1971-72, and agen- 
cies on the ETS sample that were not visited at that time, both the numbers 
of new programs that were initiated and the priorities of those programs 
were recorded. 

The source of information about initial programs for 1971-72 was the 
Mid-Year Reimbursement Record dated March 18, 1972. Information about new 
programs was recorded from Section D's of Annual Plans. These records pre- 
sented no evidence that visitation was followed by increased numbers of 
programs^ Similarly, when all data was combined* the proportions of new 
programs having priorities A, B, and C did not change between Fiscal Year 
1972 mid-year reimbursement records and Fiscal Year 1973 Section D's. 
Proportions of programs tallied by priority area, as recorded in Table 4, 
show negligible changes between the years. 

Self-Reactance of the Three Phase System 

The Three Phase System has been designed to be self-reacting. This is 
to say that its own evaluation has been designed for it and that the pur- 
pose of this evaluation is to assure the System will not fail to change for 
the better with time. Because the self-evaluation system that has been pre- 
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pared is a very extensive one this study would not be complete without spe- 
cific attention being given to it. 

The core of the self-evaluation system consists of (1) Local Districts 
One and Five Year Plans, (2) Demographic Data Forms, (3) On-site Evaluation 
Instruments (Team Leader Forms) , (4) Preliminary Evaluation Instruments for 
Students, Teachers and Coordinators, Counselors and Administrators, and (5) 
Evaluation Reports that are prepared. Data from these sources were, in 
Fiscal Year 1972, coded, stored and processed for use in a three day evalu- 
ation workshop that was conducted May 9-11, 1972 with team leaders, region- 
al directors, and other persons in attendance. Information assembled and 
reported at the meeting about follow-ups to the evaluation visits was favor- 
able to visitation in approximately the same ways that data collected for 
this specific study was favorable. For example, three-fourths of 910 
teachers and administrators who responded to the follow-up questionnaire 
sent to them by the Approval and Evaluation Unit expressed agreement with 
the evaluation report. Similarly, more than 90 percent of respondents of 
all types agreed that having members of the business conmunity on the evalu- 
ation team was a good idea. 

The broad variety of other items that was included in the question- 
naires can and does provide broad amounts of other information about the 
performance of the System. Effectiveness of -he film strip, conduct of 
team members during interview sessions, the quality of reporting, and mean- 
ingfulness of content of the report are all inquired about. 

Is there evidence that the managers of the Three Phase System are 
responding to information they collect about it? The answer to this 
question must be "yes" for lack of follow-up information about students has 
been discerned and acted upon. For Fiscal Year 1973 visitations will be 
preceded by the offering of administration of a follow-up service conducted 
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by Eastern Illinois University to agencies. Similarly, the pilot conduct 
of site visitations was followed by modification of the profile reporting 
practices in favor of the current system. Numerous other types of evi- 
dence can be cited to indicate that the System is being modified as infor- 
mation is acquired about its effectiveness. It has been developed as a 
highly reactant process. 

Effectiveness of the Three Phase System 
in Meeting Information Needs 

Table 5 was prepared to briefly summarize relationships that exist 
between Three Phase System operations and external and internal informa- 
tion requirements of the DVTE that were discussed previously. Immediately 
preceding sections of this report have given elaborative information about 
how the System performs; the chapter on recommendations that concludes 
the document contains background information related to specific action 
recommendations. Through the use of Table 5, readers are helped to com- 
pare the performance of the System against the information needs of the 
Division and to gain a simple summary understanding of its adequacy. 

Inspection of the Table 5 reveals that the System was intended to 
provide at least partial answers to all of the needs questions except the 
very first. No formal provisions were made when the System was developed 
to assure that the competencies of students at the time they completed 
their occupational program enrollments would be assessed. This is, very 
evidently, the most striking and serious deficiency of the System. It 
provides no output information! That is, it functions with the expecta- 
tion that concern for so-called process characteristics of programs will 
result in the acquisition by students of needed available skills. In 
fact, no matter how popular the Three Phase System may be with those in- 
volved in its use, it will have but slight value for determining the 
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benefits of expenditures until program products, as reflected by student 
performances, are better determined. This necessary relationship be- 
tween product followup and evaluation has been ably stated by Butler: 

The time has come to hold schools accountable for the 
success of their product. For too long vocational edu- 
cation (and all education for that matter) has been 
trying to justify itself by measuring the resources that 
go into the system rather than evaluating the product- 
the graduate. The irajority of evaluation reports still 
concentrate on comparative studies of teacher salaries, 
equipment lists, and costs, student-staff ratios, physi- 
cal plant costs, etc., paying little or no attention to 
the qualitative success of the students after they have 
left the schools.^ 

That the Three Phase System was not initially designed to acquire perfor- 
mance information means that it suffers from the common crippling evalua- 
tive deficiency noted by Butler. 

The System has, however, been designed to provide a very broad 
range of other types of information. In the third column of Table 5 
brief statements are made about the extent to which answers to needs ques- 
tions are provided by the System. In the last column equally brief state- 
ments are made about procedural changes that might help the System to bet- 
ter meet the information needs of the Division. Of course, in some cases, 
little or no change is needed. 

Improved planning for the development of a fully comprehensive in- 
formation system is clearly needed. However, it can be seen that much of 
the data now assembled by using the Three Phase System is well suited to 
serve as a basis for future system development. For example, existing 
taped records of numbers of reimbursed agencies and programs seem to be 
of high quality (External Need 2). Also, a large amount of information 



8 F. Coit Butler. Instructional Systems Development for Vocational 
and Technical Training , Englewood Cliffs, New Jersey: Educational 
Technology Publications, Inc., 1972, pp. 167-168. 
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about the effectiveness of the System actually describes both programs 
and their evaluation (Internal Need 3). Extensive software links cur- 
rently exist between data recording procedures of the System and fis- 
cal accounting practices. Indeed, as a result of efforts to make the 
System reactant an unusually high quality of information for self 
assessment is being collected; this information can be stored and 
built upon as an improved System is sought (Internal Need 3). 

The System is not designed to record information about the numbers 
and qualifications of available teachers although personnel ratings de- 
livered by the site visitation procedure have been stored. Lack of var- 
iability in this data and the questionable validity of if for the pre- 
diction of teacher effectiveness in imparting job entry skills suggest 
however, that if assessment of teacher qualifications should be an im- 
portant part of an evaluation system for the DVTE much work remains to 
be done to make the present System effective for this purpose. 

A variety of types of information have previously been cited to 
suggest that the information LEA's are receiving is importantly defi- 
cient for the purpose of improving instructional offerings (External 
Need 4). Local agencies lack information about community and student 
needs although some such information will be provided Tor agencies 
using the follow-up service that is being pilot tested in Fiscal Year 
1973. It is possible that no state possesses a strong needs assess- 
ment service and the Three Phase System should not be excessively 
faulted for this reason. However, this deficiency must be recognized 
as long range planning for improvement is conducted. 

The System seems to be providing good information about inade- 
quacies of guidance personnel and services (External Need 5). 
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Deficiencies have clearly been documented through use of the System, The 
deficiencies that have been documented related also to needs for place-* 
ment services (External Need 7). It is certain that the System could be 
improved by increased concern for the collection of information about job 
placement; emphasis in this effort should include concern for students 
who are not college bound and dropouts. Very greatly expanded follow-up 
studies are thus called for. 

This study identified one problem which the division cannot respond 
to by itself. The problem stems from the fact that the Three Phase Sys- 
tem acquires information about enrollments in courses but this informa- 
tion, because it does not provide an unduplicated count, cannot be used 
for federal reporting purposes. Conseqv utly , C:? Fiscal and Statistical 
Unit finds it necessary to conduct a special survey of agencies to ac- 
quire an unduplicated count for that purpose. Somehow, a continued ef- 
fort should be maintained to assure thac either federal or state prac- 
tices be changed in a manner which will permit one type of reporting by 
local agencies for both purposes. 

This study uncovered no formal procedures for the transmission of in- 
formation pertinent to internal Division decision making to its general 
management staff. This statement should not be interpreted to mean that 
that no evidence of action upon information can be recognized. On the 
contrary, the development of a handbook for the preparation of behavior- 
al objectives and other activities attest to the fact that feedback oc- 
curs and action is taken. However, until the outcomes of application of 
the Three Phase System are given formal consideration for priority set- 
ting (Internal fteed 1) and other purposes it is unlikely it could ac- 
quire full effectiveness. 
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Accountability for expenditures is partially met by the System as 
augmented by audits of the Fiscal and Statistical Unit. However, as 
noted earlier these audits are not now current nor do they proceed be- 
yond the level attainable by concern for pupil registration and record 
keeping. Therefore, until audits can be updated and until they include 
the Chicago Public Schools it is clear the System will remain deficient 
in this sphere of responsibility. 

The System is intended to facilitate and coordinate the delivery 
of consultation services to agencies (Internal Need 4). The sincerity 
of Division personnel within the Program Approval and Evaluation Unit 
in their personal efforts to be of help is amply documented. Similarly, 
a large amount of consultation is provided by the Division, The consul- 
tation that is currently provided through contributions of the Three 
Phase System are at least minimally adequate but evidence was not ac- 
quired that strongly supports the position consultative support to local 
agencies is highly effective. This needs additional attention. 

The System delivers a substantial amount of information about the 
involvement of community members in program development but this is 
clearly an area of involvement that will demand continuous attention 
(Internal Need 5). Community involvement is closely linked with supply 
and demand information requirements for the state which is, in turn, a 
need that is not well met (Internal Need 6). The System must either be 
drastically redesigned or heavily augmented if supply and demand infor- 
mation is to be efficiently assembled. 

The strengths and weaknesses that have just been cited are, from 
the effort involved in their assembly, prepared so that a totally favor- 
able statement of System effectiveness cannot be made. Therefor % it 
seems important t<> strongly emphasize that the study has identified 
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staff and System reactance to be its greatest strength. If one were to 
select a preferred strength for a System it might be this. The System 
has been developed with major concern for detail that will permit its 
own systematic evaluation and improvement. Its staff seem highly open 
to criticism and suggestions which implies they will not reject sound 
advice that is given to them. Adaptation of the System is clearly 
called for because of the severe deficiencies imposed by its lack of 
output information. This deficiency is one that the Three Phast' System 
has in common with systems (or nonsystems) for state level evaluation 
that are used in other states. Immediate total revision of the System 
would therefore be unwise; long range planning is needed to assure 
that it will not remain in its present form beyond a time when major 
improvements can be designed and made* 

Evaluation in Other States 

Other states have also been called upon to undertake comprehensive 
evaluation of vocational and technical education programs. Ic goes with- 
out saying that evaluation and accountability are receiving considerable 
attention in all state* vocational-technical education groups. A most 
significant recent account bility event has been Marland f s Revision of 
Federal Regulations, dated January 20, 1972, which requires annual eval- 
uation reports by State Advisory Councils and thereby further establishes 
the role of Advisory Councils in accountability relationships. 

An important work in the quest for systematizing the statewide • 
evaluation is that of Starr, Dief f enderf er , Archer and Ernst, A System 
for State Evaluation of Vocational Education (1970). 9 

9 Harold Starr, Richard A. Dief f enderf er, B. B. Archer, and Mireille Ernst, 
A System for State Evaluation of Vocational Education Columbus, Ohio: The 
Center for Vocational and Technical Education, The Ohio State University, 1970. 
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This comprehensive document discusses the logic and logistics of evalu- 
ation from the intended purpose and operational characteristics of the 
desired system through system design strategy to an organizational pat- 
tern for staff. 

The Center for Vocational and Technical Education effort initially 
involved three states — Colorado, Kentucky, and New Jersey. Two other 
states — Connecticut and Michigan — later participated in the developmental 
work, and the system was field tested in Michigan and Colorado. 

Other states have been moving toward statewide system-oriented eval- 
uation of v^ational education. In 1969, the Pennsylvania Research Co- 
ordinating Unit for Vocational Education sponsored a comprehensive review 
of the vocational education services in that state. 10 In the following 
year, a statewide evaluation of vocational education programs was carried 
out for the Pennsylvania State Advisory Council for Vocational Education 
(An Evaluation of Vocational-Technical Education in Pennsylvania, 1970 ) . 1 1 
This last effort seems to be of the sort envisioned by the U.S. Commissioner 
of Education Mar land in his recent alteration of federal regulations. 

Most writers stress the importance of continuous, as opposed to peri- 
odic, evaluation. An example of a statewide system for evaluation which 
involves a continuing effort — and a continuing refinement of evaluative 
technique — may be found in Oklahoma. The objectives of the Oklahoma project 



AU Walter Arnold, Vocational Technical and Continuing Education in Pennsyl- 
vania; A Systems Approach to State-Local Program Planning Karrisburg : Pennsyl- 
vania Research Coordinating Unit for Vocational Education, 1969. 

n An Evaluation of Vocational-Technical Education in Pennsylvania , Prepared 
for the State Advisory Council for Vocational Education Washington, L.C.: 
American Vocational Research Corporation, 1970. 
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are clearly stated: 



The overall purpose of this project was to develop and 
initiate a continuous and detailed .Occupational Training 
Information System (OTIS) to provide a better data base 
for encouraging necessary changes in Oklahoma's State 
Plan for Vocational Education and in consequent patterns 
of Occupational offerings and enrollments. 12 

What should be particularly noted here is the emphasis placed upon on- 
going utilization of evaluation results in program revision. This has of- 
ten been a weak area in evaluation schemes. Oklahoma's commitment to fur- 
ther work here may be seen in a description of one of the OTIS activities- 
the "Development of a State-Directed Evaluation System for Vocational and 

Technical Education in Oklahoma": 

This project will concern itself with instrument designs, 
objectives of the system, retrieval procedures, analysis 
. of local programs in meeting their objectives, dissemina- 
tion of findings to the state and local level for program 
planning and improvement.^ 

Other states are moving in this direction. Massachusetts has what is 
called an "Information Feedback System," which involves evaluation on a num- 
ber of levels - Program Evaluation, process-product evaluation, cost- 
effectiveness evaluation, an^ overall evaluation of vocational-technical 
education. Two aspects of the Massachusetts system are of particular 
interest. The first is the attempt to establish a system whereby eval-* 
uation may be effective without destroying local autonomy. The second is 



l2 Research and Research-Related Activities of the Division of Research , 
Planning and Evaluation - 1970-71 Stillwater, Oklahoma: Vocational Research 
Coordinating Unit, Oklahoma State University, 1972. 

13 Ibid. 

14 Kathryn H. Speiss and Eugene R. Speiss (eds.) A Guide to Evaluation : 
Massachusetts Informatio n Fe e dback System for Vocational Education. First 
Technical Progress Report Boston: Massachusetts Research Coordinating Unit 
for Vocational Education, 1969. 
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the concern not only with allocation of effort in vocational education 
planning, but with the place of vocational education within the larger 
educational system. 

This is reflected in Massachusetts development of a statewide system 
of evaluation for all education* The -evaluation system, like the Informa- 
tion Feedback Systes, involves a cooperative approach with local education 
agencies. 

The Massachusetts system involves the following steps: 

1) Identification of needs of chilren, youth, and society in general. 

2) Establishment of goals for education consistent with these identi- 
fication needs. 

3) Development of assessment and evaluation activities to measure 
learner achievement and program success in terms of the goals. 

4) Repetition of the above three step cycle through the redefini- 
tion of goals and the revision of programs. 15 

While not specifically concerned with vocational education, the Massachu- 
setts statewide system of evaluation of education serves as a source for 
discussion of statewide evaluation schemes. It also at least raises the 
issue of compatition for education dollars between vocational education and 
other education. The usual approach here has been cost-effectiveness or 
cost-benefit analysis. 

According to Voelkner and Ollis, 16 Texas and Michigan have attempted 
to calculate the comparative costs of vocational and secondary education. 
There were methodological problems with this approach, which may be over- 
come through a greater refinement of technique. The Michigan State Advi- 
sory Council has funded further work on this problem. 

A Report of Local Educational Agency on Educational Goals for Massachusetts 
Boston: Commonwealth of Massachusetts, Department of Education. 1972. 
1 6 

Alvin R. Voelkner and Harvey X. Ollis, The Economic and Non-Economic 
Evaluation of Vocational Education 1971. 
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Another source of guidance for developing a statewide evaluation plan 
is Nerden's article on the subject, which appears in Contemporary Concepts 
in Vocational Education : The First Yearbook of the American Vocational 
Association * * 7 

Nerden presents a chronology of evaluation that gives special empha- 
sis to the use of outside resources in the evaluative process. 

1. Consulting Committee 

First, there is the very obvious urgency that each state director or 
leader of the statewide vocational education activity assemble, in keeping 
with sound vocational education practice, a consulting committee concerned 
with all aspects of program evaluation. The responsibility of this com- 
mittee should not be confused with those of the State Advisory Council on 
Vocational Education. 

2. Broad Representation Outside of Education 

Employers, representatives of labor and management should also be in- 
cluded in the evaluation consulting committee. 

3. Regional and Local Involvement 

Once the consulting committee has made some tentative decisions con- 
cerning how best to proceed, every effort should be made to move operation- 
al aspects of the activity to the regional and local situations. Efforts 
to involve all teachers, local supervisors and local administrators in the 
evaluative process and specifically in the development, try out, and sub- 
sequent modification of guidelines, criteria and procedures. 



l7 Joseph Nerden, Statewide Evaluation of Vocational Education, 
Contemporary Concepts in Vocational Education : The First Yearbook of the 
American Vocational Association , ed. Gordon F. Law Washington, D.C.: 
American Vocational Association, 1971. 
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4. Synthesizing Results 

Assuming that every individual involved in vocational education has 
had ample opportunity to participate in the development, preparation 
and/or modification of guidelines, criteria and procedures to be used in 
evaluation, the next major step is to assemble the collective judgments of 
the regions and areas of the state and to synthesize the results so that 
they represent consensus as nearly as possible and eliminate duplication 
and inconsistencies* Throughout the statewide activity, domination or re- 
strictive influence by State Department of Education supervisory or admin- 
istrative staff must be avoided. By keeping its involvement strictly lim- 
ited to leadership, assistance and advice, the State staff may best assure 
itself of a successful venture in evaluat r on. 

5. Setting Priorities and Providing for Change 

Following the development of valid a*id reliable instruments for eval- 
uating vocational education, a variety of self study, visitation, collab- 
oration, priority setting and follow-up activities must take place. 

A conference report of the National Institute on Improving Vocational 
Education Evaluation makes the following recommendations for tne improve- 
ment of evaluation: 

1. It is urgent that state departments and local school systems 
begin to cooperatively and carefully plan for the collection 
of data needed to evaluate existing programs and plan new ones. 

2. It is equally urgent that persons at all levels seek to improve 
available evaluative techniques as well as to develop new and 
better ones. 

3. Evaluation efforts need to be coordinated in order to avoid 
unnecessary duplication. 

4. The implications of the 1968 Amendments for evaluation are 
many and complex requiring that we learn and implement those 
procedures and techniques that will most efficiently aid the 
decision making process for the attainment of local, state 
and national objective. 
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5. Evaluation is a challenge to all persons involved in vocational 
education. 18 

These recommendations have been adhered to in the development of the 
Three Phase System; but if one concludes, as is appropriate, that the 
development of a fully comprehensive system has not been achieved in 
Illinois, the recommendations are still pertinent. 

Summary 

This investigation revealed strong support for continued use of the 
Three Phase System. Both educators and non-educators judged the System to 
be valuable and worth the effort that it requires. An important aspect of 
the System is that it is designed to produce and deliver large quantities 
of information that is useful and necessary for its own modification; it 
can for this reason be rated highly self-reactant and capable of modifica- 
tion. It has produced changes in local plans from one year to the next, 
and possesses a functional feedback loop that involves the Program Approval 
and Evaluation Unit, other units of the DVTE, and local education agencies. 
Therefore, a major strength of the System is that information is periodi- 
cally obtained that can permit the evolution of a more effective form of 
evaluation. 

But the System produces discrepant information. In the composite 
ratings that are prepared from site visitations, agency personnel receive 
consistently high ratings at the same time that Resources Utilized, which 
must be linked to personnel performance, is negatively related to the 

* 8 Guide to Improving Vocational Education Evaluation, Summary of the 
National Institute on Improving Vocational Education Evaluation on 
August 4-8, 1969 (Fayetteville, Arkansas: Arkansas University, College 
of Education, 1970). 
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financial needs of agencies; this suggests that a serious personnel defi- 
ciency may go unrecognized by visitation teams. A consistently negative 
relationship between financial needs and ratings suggests that, if com- 
posite ratings reflect "local potential achievement", proportionately 
higher support should perhaps be given to poorer agencies than is currently 
being allocated. Of course, since no information exists to establish that 
local potential is realistically considered when ratings are made, this 
conclusion cannot be strongly made. Indeed, because of lack of information 
about how these adjustments are made by visitation teams, composite 
ratings have limited established validity. 

Planning activities of agencies have numerous deficiencies* Some 
deficiencies may arise in part because documents prepared by the Division 

to guide planners lack the kinds of specificity and elaboration that 

would make planning easier. However, other types of assistance are also 

needed. Half or more of the respondents to one of the questionnaires 

distributed in this study agreed or strongly agreed that their planning 

activities would have been more successful had they been given additional 

external help to: 

1. Conduct student follow-up studies. 

2. Conduct needs surveys in their communities. 
3* Conduct employer need surveys. 

4. Prepare behavioral objectives. 

5. Conduct effective evaluations. 

6. Identify job entry skills of students. 

7. Set priorities for program development. 

The Division currently provides some assistance in each of these areas 
and cannot be faulted for not being fully helpful in each, but it is obvious 
that a very great amount of assistance is needed by local agencies if they 
are to become more effective in their own evaluation efforts. In particular, 
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they will need assistance in skills assessment if comprehensive statewide 
evaluation is to be highly effective. Their other needs for assistance 
link to, and are generally dependent upon, improved skill assessment if 
they are to be met. 

Without a large amount of output information, accountability is 
not possible and evaluation must be judged to be seriously defective. 
Accountability requirements, in the case of occupational programs, call 
for intensive attention to coordinated work placement efforts. These 
efforts are to be built upon product performance feedback to educational 
agencies. The feedback that is required should describe, among other 
things, how the skills of program graduates related to their work settings 
A large amount of this type of output information is basic for accounta- 
bility requirements. 

The Three Phase System, as developed and implemented in Illinois, 
does not emphasize assessment of the output of programs. Instead, under 
the tacit assumption that if conditions are present outcomes will result, 
the System concerns itself with the contents of plans submitted by agen- 
cies and what can be learned from site visitations keyed to traditional 
process concerns. These types of information cannot, however popular the 
System may be at this time, firmly establish the value of programs to 
their graduates. Without assessment of the skills graduates possess and 
need to use when employed, this and other similar evaluation systems must 
be judged to be critically deficient. 

Development of a fully comprehensive evaluation system has not been 
achieved in Illinois. Cooperation between the different agencies is still 
needed and coordination is required. This coordination should, it must be 
noted, extend beyond the boundaries of individual states and incorporate 
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concern for what states can do together. Therefore, just as the Illinois 
System has already benefited from information developed elsewhere £'id is 
currently inviting future multi-state efforts, a major need for future 
comprehensive system development in Illinois cannot be met without formal 
attention given to that need. 19 



19 A Request for Proposal was issued by the Research and Development 
Unit for development and testing of a "Systematic Approach for Implemen- 
tation of a Statewide, Student Based, Management Information System." The 
time frame for this development extends from November 1, 1972 to June 30, 
1973. 
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RECOMMENDATIONS 



It has become common practice for persons making recommendations for the 
development of evaluation procedures to advocate that a complete systems ap- 
proach be applied during their preparation. Indeed, if such, an approach to 
evaluation is not taken, some writers are inclined to strongly condemn what is 
being done and to label it grossly or totally inadequate. This tendency in 
current thinking about evaluation, when used in company with earlier precedents 
of labeling evaluations that do not assess program outputs as useless, can lead 
to totally negative criticism of any evaluation procedures that are adopted for 
present day use. If these standards are the only ones applied to twe Three 
Phase System, it would be necessary to conclude that it is suitable for some 
purposes of local planning but of no substantial value as a statewide evalua- 
tion device. 

Dyer has recognized deficiencies of present day evaluative systems and 
suggested a position from which to review them. He stated: 



One could go on endlessly elaborating the kinds of infor- 
mation required to make an educational system genuinely respon- 
sive to the needs of the students and the times, present and 
future, and thereby one would demonstrate beyond a doubt that 
the whole id^a is fantastic and wildly unrealistic and completely 
beyond the 1m of human possibility. Of course such complex 
varieties of .eedback are unavailable to us now; the kinds of 
instrumentation necessary to make them available, with respect 
to both inputs and outputs, are still crude despite the work 
that has gone into them over the last 50 years or so. But 
this is no reason for not trying to imagine what an ideal feed- 
back system in education would be like and for doing vhacever 
can be done to realize it in practice as best we can. 1 



A Henry S. Dyer, The Role of Evaluation in School Systems. Talk given to 
New Jersey Association of School Administrators, September 22, 1971 (Reprint: 
Educational Testing Service). 
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This report has been prepared with the intention of avoiding excessively 
negative criticism based upon theoretical considerations for which there are 
now no expedient means of implementation. For example, proven competency- 
based assessment devices for statewide evaluation of student performance do 
not now exist. Nor has any state or local education agency successfully de- 
veloped and installed a fully comprehensive system! Therefore, while it is 
necessary to keep firmly in mind that the Three Phase System is critically 
deficient when compared against idealized standards and to avoid encouraging 
its users or those who learn about it to conclude it cari be retained or 
adopted without need for constant revision and adjustment, the system must 
not be excessively criticized and condemned. Both strengths and deficiencies 
of the system must be recognized with the int ntion that its strengths be re- 
tained and remedies sought for its deficiencies through an evolutionary pro- 
cess. More ideal evaluative arrangements should be sought as rapidly as they 
can be developed. The recommendations of this report are presented with the 
intention that they may provide useful direction toward short and long range 
evolutionary development of a more ideal system. 

The effectiveness and efficiency of an evaluation system are importantly 
dependent on the accuracy and completeness of statements of goals and objec- 
tives of the organization being evaluated. Statements of goals and objectives 
in turn largely determine the information requirements to be met by an effec- 
tive evaluation system. It was noted previously that statements of objectives 
for the Division (whether directly referred to as objectives or noted as "goals, " 
"purposes, 11 or otherwise) were so numerous as to be impossible to work with ex- 
cept in a summary condensation. A total of thirteen information needs were sub- 
sequently prepared against which the effectiveness of the Three Phase System 
might be judged. The recomnendations of this report have been prepared as a 
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result of crosschecks of the performance of the Three Phase Syste yn against 
the summarized information needs of the Division. They are organized into 
groups on the basis of whether or not the action that would be required would 
be or long or short duration and are presented with additional remarks de- 
signt J to clarify their intended meaning. 

Long Range Recommendations 
The Three Phase System has as its major deficiency lack of information 
about job entry performances of students who complete occupational programs. 
This weakness exists at the center of a cluster of closely related information 
deficiencies all of which urgently require attention if statewide evaluation 
is to be made comprehensive. What jobs are available? What placement succes- 
ses have students had? What skills are needed to perform well on jobs? How 
satisfied are students and employers with the preparation provided in occupa- 
tional programs? All of these questions have concern for the outputs of pro- 
grams. Lack of success at assessing outcomes is the primary deficiency of the 
Three Phase System. Ultimately, evidence of program effectiveness must be 
sought and reasons for program planning and modification must be assembled. 
This will require long range development in the three areas that follow. 

1. A four-year plan for performance assessment should be initiated. 

The primary deficiency of the Three Phase System is that it has lacked 
really productive concern for job entry skill assessment and for the ques- 
tions which relate to it. A very good reason for not having an operational 
performance assessment program is that its development costs would have far 
exceeded those for the Three Phase System. However, at the same time that 
some. task analysis projects have been initiated in Illinois (the Lincoln 
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Land Community College project is illust" * v » # other skill assessment 

projects have been initiated about the c . ^ry such as Project Career in Mas- 
sachusetts. Conceptual work by Loch 3 which was '^scribed earlier placed this 
type of assessment within a broad framework fo .ehensive evaluation; work 

by Thornton and Wasdyke 1 * on criterion rcferenceo ;ing also shows good prom- 
ise for effective development of skill assessment practices. 

The selection of a four year cycle for performance development assess- 
ment by the DVTE is arbitrary but meant to make Known the amount of time that 
may be required to develop such an approach. Provus, 5 in Discrepancy Evaluation 
* broke a typical assessment down into the following approximate levels wMch, 
when applied in Pittsburgh, Pennsylvania, and other cities, took about one year 
per level or stage: 

Level One: Educational Goal Definition 

Level Two: Translation of Goals to Measurement Objectives; development 

and/or collection of applicable instruments 
Level Three: Data Collection 

Level Four: Statement of Needs (Discrepancy existing between Goals 
and Student Assessment Performance), 

The levels used by Provus may not be equally suitable to Illinois but it is 
important to note that the four year time interval was partially tested in 
these efforts. 



2 John G. Nee, The Identification and Comparison of the Tasks for the 
Occupational Role of Industrial Production Technologist » Lincoln Land 
Community College, Unpublished paper, July 1972. 

^Charles Loch, Model //3 - Vocational Education: Emphasis on Career 
Development. Memorandum dated September 23, 1971. 

^Richard F. Thornton and Raymond G. Wasdyke, The Development of 
Criterion ReferencedTests Using the Behavioral Consistency Model . TDM- 72- 
August, 1972: Educational Testing Service, Princeton, New Jersey. 

5 Malcom Provus, Discrepancy Evaluation for Educational Program Improve- 
ment and Assessment . Berkeley, California: McCutchan Publishing Corp., 1971. 
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The evolutionary development of a new output-focused evaluation system 
for the Division might, it must be noted, follow a.*»ong lines where site visi- 
tation becomes more and more keyed to output assessment. For example, visita- 
tion teams might over time become involved in the field trial of performance 
assessment studies. It is unlikely that this approach should be attempted on 
more than one or two curricula for which a large student population is involved 
and assessment is judged most feasible. Certainly, it would not be possible 
to initiate this approach in all occupational areas at one time. 

Equally certain is that this type of development would require collab- 
oration with external agencies. It is for this reason that establishment 
of a Technical Advisory Committee is recommended as part of the short range 
actions in the next section. No single state can afford the expense of de- 
veloping this type of system. Indeed, even a consortium of states and agen- 
cies will have difficulty doing so. Therefore, it is critically important 
that the Division seek to avail itself of all information and opportunities 
open to it. Collaborative development with other agencies seems to be the 
only approach that can keep costs within the limits that exist in Illinois. 
The development of the National Occupational Competency Testing Program, 
which itself has potential value for the determination of the competency 
of teachers illustrates the benefits to be found in cooperative development 
programs.* 

2. A continuous effort should be made to improve supply -and-demand studies. 

The Three Phase System hao been designed with the assumption that local edu- 
cation agencies should undertake much of the work necessary to determine what 
opportunities will exist for its students and to achieve a fit between the 

opportunities that exist and the educational programs designed for them. It 
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*This project is now in its developmental phase; Carl Schaefer of 
Rutgers State University of New Jersey is currently project director. 
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is true that local agencies are expected to use information given them by the 
employment services and other groups. But the information those groups have 
provided to date is not good enough to permit the local agency to closely tune 
its curriculum to opportunities. 

This problem is also one that is too great for ?ocal agencies and, perhaps, 
even for states to meet. Changes in employment patterns in Chicago and its sub- 
urbs have, for example, recently resulted in a condition where the dominant 
transportation pattern ror work has switched from a suburb-to-city flow to a 
city-to-suburb flow. It can easily be seen that such forces as industrial 
plant development, changes in consumer demand, and national and international 
economic development are far beyond the capabilities of restricted areas to 
understand or sense. These forces produce demands for regional, state and 
national study. They also require high levels of cooperation among agencies 
and competent long range planning. Clearly, the potential effectiveness of 
the Three Phase System has been severely restricted by lack of this type of 
information. 



3. Formal, continuing efforts should be made to improve comprehensive 
evaluation and development. 
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Comprehensive planning for statewide evaluation of programs may be ini- 
tiated through the use of a half-dozen or more guidelines. The guidelines will 
seem idealized and impossible of attainment in the short range. Indeed, they 
may be so general as to discourage some groups from attempting to approach them. 
However, their intended purpose is not to set standards for immediate attain- 
ment but rather to enable planners to establish long range goals some of which 
will be easier to attain than others but all of which will require persistent 
attention over years. Six guidelines follow; each is accompanied by a brief 
description of why it is included: 



Guideline I ; Mechanism for Establishing Statewide Goals and Priorities for 
Vocational-Technical Education 

Responsbility for establishing statewide goals is shared by many groups 
within the state. Any differences in either philosophical or programmatic 
goals should be resolved as part of initiation of a system of statewide eval- 
uation. 

Guideline II ; Adm inistrative Organization for Statewide Evaluation 

If evaluation is to be a continuous process, it must be built into the 
fabric of the state administrative structure. This means that an appropriate 
staff organization with funds necessary to function must be provided. 
Guidel ^ e III : Budget Provisions for Statewide Evaluation on a Continuing Basis 

Since evaluation should be a continuing process, not a one-shot endeavor, 
budget provisions should be made on a continuing basis. 
Guideline I\ : Comprehensive Plan for the Evaluative Process 

A comprehensive plan for statewide evaluation should show (a) major and 
minor objectives; (b) concern for all institutions, agencies and offices in- 
volved 5 (c) proposed evaluation processes and procedures, complete with a de- 
tailed accounting of the purposes for evaluation components; (d) the way each 
evaluation component will work; and (e) in plain words, how evaluation data 
will be used. 

Guideline V : Feedback Mechanism for Quickly Identifying Problem Areas 

If we accept the idea that good evaluation is a dynamic, continuing process, 
it is imperative that feedback mechanisms for improving both the evaluation sys- 
tem and the statewide vocational technical education system it purports to ana- 
lyze are incorporated into the system. It must be clear that major objectives 
should not be subject to coistant change. However, the various indicators, pro- 
cesses and criteria for measuring such elements as quality, accessibility, di- 
versity, provisions for disadvantaged, and opportunities for career development 



as a life-long process are the life-blood of evaluation; they should not 
remain static. 

Guideline VI : Administrative Mechanism for Analyzing Data and Affecting 
Necessary Changes 

The worst thing that can happen to a carefully developed evaluation report 
is to have its recommendations ignored or lost in bureaucratic processes • When- 
ever an evaluation system is designed, it is very important that the results of 
the evaluation will be transformed into specific plans of action by specific 
offices of the State and by Local Education Agencies. At the present tinn 
the Three Phase System does not have input in the total planning system for 
Vocational-Technical Education in Illinois. This guideline is therefore par- 
ticularly important for this state. 

Short Range Recommendations 

Both long and short range recommendations have been prepared with the 
assumption that implementation would occur within a half year. The differ- 
ence is that short range recommendations may be met within a year while 
long range recommendations cannot possibly be attained in that time. 

2. Establish a technical advisory panel to advise on the evolution 
of a move output-centered system. The committee might have a 
half-dozen members with most from out of state and involved in 
state level evaluation. The charge of the committee would be 
to recommend changes in tne current system through research 
and development. 

The purpose of an advisory panel, as thought of in this report, is to 
formally provide for external guidance and long range development. The devel- 
opment of competency-based evaluation, improved supply and demand studies and 
adoption of comprehensive evaluation practices emphasizing output should not 
be undertaken without outside consultation. The consultation should be inde- 
pendent of any existing power structure or vested interest and should be of 
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national stature. Its members should be selected for the specific knowledge 
they hold about comprehensive assessment problems faced by the Division. Five 
or six members might be sufficient with a rotating membership in use over the 
years. Rotation would be preferred so that new members (three per year on a 
six member group) could be brought in as new technical problems are identified. 

Four meetings might be appropriate the first year. The charge that would 
be given to members when first brought together would be to address themselves 
to the task of advising for successful initiation of long term actions of the 
preceding section. A first meeting might cover three days after which subse- 
quent meetings might take less time. The group would select its own chairman 
with the understanding that 2 of 5 or 3 of 6 members would be r^laced at the 
conclusion of the first year b> individuals selected for specialized knowledge 
needed in the second year. No member would be retained beyond the three year 
interval in order to assure that the committee would both retain capacity for 
renewal and continuity in its efforts. 

2. Fiscal audits should be rapidly pursued so that agencies will 
no longer experience as many discrepancies as have appealed 
in the past. 

Fiscal audits ha ,r e lagged well behind current operations. The type of 
audit that is undertaken is not sufficient to establish financial accountability 
for the pooling of resources toward needy areas but it has served to make Agen- 
cies aware of their responubilities for financial accountability and has re- 
vealed that procedures in use prior to the adoption of the Three Phase System 
were inadequate. Therefore, this activity should be continued and speeded up, 
although this recommendation should not be construed to mean that all agencies 
should be audited each year. On the contrary, because some agencies are very 
small this activity would not be cost beneficial. A cost beneficial but fully 
current audit should be sought. 
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3. Profile analyses should be conducted for each new round of 
evaluation reports. Evidence of low variability and lack 
of discrimination should be sought in these analyse^. 

It was noted in this study that composite rating'; for personnel lacked 
variability; no evidence arose from analyses of profile data to suggest 
systematic biases were induced by team leaders. Strengths and weaknesses of 
the composite rating system are partially assessable by the types of analyses 
that were undertaken for this study. These analyses should be repeated and 
improved over time. 

It can be seen that the rating currently given to personnel may have lit- 
tle validity (what does it mean for 85 percent to be rated "above average"?) 
as an indicator of program effectiveness and may serve largely as a means for 
complimenting staff members and thus assuring acceptance of the visitation 
s am. Unless this and other rating categories possess variability they 
shoula be eliminated from further use. 

Also, it is important to continue to be alert to the possibility that 
the visitation system may experience deterioration in its existing effective- 
ness because of cont^aued use of visitors who live near to the agency being 
visited. Profile analysis can give clues to the possibility this is happening. 
It can also regularly reveal relationships among ratings for different types 
of agencies, sizes of teams visiting them, and other variables. 

4. The follow-up of students who have been enrolled in programs 
should receive greatly expanded emphasis; follow-up should 
be closely tied to the Three Phase System. 

The Three Phase System has been operated from an underlying assumption that 
if certain conditions judged to be important for an effective program are pres- 
ent the output from the program will be satisfactory. The presence cf qualified 
teachers, up-to-date equipment, a good curriculum, et cetera would, it has been 
assumed, produce graduates who have marketable entry level skills and who will 
make desirable employees, while schools lacking in these attributes will produce 
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less competent and less desirable employees. Actually, without extensive 
follow-up of students there is no way of knowing whether there is any 
important relationship between the process variables and the real life cri- 
terion of m-the-job effectiveness. Therefore, it is important to know 
what happens to students (preferably all of them) after they leave school 
as dropouts or with ^plomas. Follow-up should provide information for at 
least three and preferably five years from course completion. 

Follow-up instrumentation should ask students to e/aluate their educa- 
tional experience. They should be asked to identify skills they found most 
valuable and to indicate how effective their schooling was in teaching them. 
Their supervisors should be asked for evaluations of the performance of for- 
mer students, especially regarding areas in which their training was inade- 
quate. Employers should also be asked to describe the work habits and atti- 
tudes of former students as they compare with those desired in responsible 
workers. C ^ction of this information should, if at all possible, be made 
an integral, mane. ->ry feature of evaluation w.th summary information pro- 
vided to the Advisory Council, the state agencies, and individual schools. 

The follow-up procedures that were prepared for pilot use by the Center 
for Educational Studies at Eastern Illinois University include many of the 
features that should be found in good follow-up instrumentation. However, 
it is not enough just to produce potentially useful items and procedures. 
The audiences to whom the instruments are addressed must include both stu- 
dents who have completed programs and those who have not. Furthermore, it 
is necessary to carry out special studies if overall response rates are 
poor or if some types of students are more difficult to follow than others. 
Clearly, this service is one that should be sponsored at the state level. 



83 



5. Improvements in the software of the system should be sought. 
In particular, the instructional sections of the Criteria and 
Guidelinea documents should be considered for simplified 
revision. 

The versions of these documents that were in use during Fiscal Year 1972 
were brief and could not be easily understood or used. Descriptions of tha 
allocation of funds are, for example, difficult to fully interpret. This is 
partly caused by use of two regular scales to describe reimbursement practices 
which actually followed a single irregular scale. Improvement of these docu- 
ments will probably require close working relationships with local agencies. 

6. A strenuous effort should be made to overcome the unduplicated- 
duplicated count problem of student accounting. 

It should be possible to adopt, modify or develop a procedure whereby edu- 
cational agencies would be able to complete "nit one set of records, as for ex- 
ample course registration cards from which both state and federal reporting 
requirements could be met. If necessary, alterations should be sought in state 
and federal reporting requirements. 

7. More formal procedures should be sought for decision-making 
within the DVTE so that evaluative information can be better 
used. 

Management procedures of the Division have not formally provided for in- 
clusion of Program Approval and Evaluation Unit information in the decision- 
making activities of the Division. For an evaluation system to be comprahen- 
sive and effective it is necessary that it be formally designed to formally 
respond to che information needs of the Division. A fully comprehensive sys- 
tem cannot be developed without this type of involvement at the Division level 
nor can existing strengths of the present system be most effectively used with- 
out it. Of course, the Administrative Planning Council of the Division has 
been the location where formal consideration ct the role evaluation might ha/e 
been expected. 
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Summary 



The Illinois Three Phase System should be temporarily retained as an 
administrative tool to be used in relationships between the Division of 
Vocational and Technical Education and local education agencies. But it 
is clear that such a system, with its scope directed toward local school 
districts and with its lack of outcome assessment and related state prior- 
ities, should not be considered as a fully satisfactory vehicle for state- 
wide evaluation. It is not necessary to do additional research to recog- 
nize that the greatest need for improvement and modification of the System 
is for improved, fully comprehensive follow-up procedures. Therefore, 
there is great need for progress in the area of performance assessment for 
students who complete programs. Because the Three Phase System as it has 
been developed and implemented is in many respects a model of how to get 
the greatest good from visitation procedures, and because the individuals 
responsible for its use have actively sought and built into its design 
procedures by which it can be changed, one cannot recommend that current 
uses of the System be suddenly and markedly changed. A supplemental devel- 
opment program is needed. The development program should lead to progres- 
sive modification of the present System with the goal of ultimately re- 
placing it with a more comprehensive output-focused system. Improved per- 
formance assessment and follow-up should be the cornerstones of a modified 
system. 
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APPENDIX A 



SEQUENTIAL DEVELOPMENT OF THE THREE PHASE SYSTEM FOR STATEWIDE 
EQUATION OF OCCUPATIONAL EDUCATION PROGRAMS 

The following outline is a brief time table of activities 
that occurred in the development of the Three Phase System for State 
wide Evaluation of Occupational Education Programs with the primary 
emphasis on Phase Three from July 1970 through May 1972. 

1969 - DECEMBER 

Workshop on Refining Phase I (1 and 5 year Local Plan for 
Vocational-Technical Education). 

1970 - JANUARY TO MAY 

Three Phase System for Evaluation formulated and directions 
established for further development. 

JULY 

Workshop to receive suggestions and recommendations from 
Local Agency personnel in developing Pha$> hree (on-site 
evaluation) . 

AUGUST - OCTOBER 

a. Developed objectives utilizing Delphi Technique (Rand 
Corporation). Participating Groups: 

1. Business and Industry 

2. Educators 

b. Phase Three specifics drafted. 
NOVEMBER 

Reaction workshop to receive assistance in formalizing 
specifics of Phase Three 

1971 - JANUARY THROUGH MAY 

Field test of Phase Three 

Sites: Mattoon High School 
Centralia High School 
Valmeyer High School 
Pekin Area Vocational Center 
Joliet Jr. College 
Orion High School 
Proviso Township High Schools 
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JULY 

Reaction wcr-cshop for Field Test participants to finalize 
Phase Three for full implementation during pilot year 1971-72. 

AUGUST 

Develop Evaluation ma trials 
SEPTEMBER 

Workshop for on-the-job training of team leaders. 

1971- 

1972- OCTOBER - MARCH 

Conduct 72 Local district evaluations and monitor the Three 
Phase System through followup of team members and districts 
evaluated, 

APRIL 

Tabulate evaluation data 
MAY 

Workshop for representative team members to interpet evaluation 
data gathered and discuss recommended changes for System refine- 
ment. 

JULY THROUGH AUGUST 

Refine the System for the 1972-73 school year. 
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APPENDIX B 



ON-SITE EVALUATION INSTRUMENT 
(Team Leader Form) 



1. What are some specific strengths of the administrative organization as 
it relates to occupational education (Cite several strengths) 

2. What ar^. some specific weaknesses of the administrative organization? 

3. How many years has the present organizational strucuuc^ Heen in existence? 

years. 

4. How would you (as a team) rate the extent of the dissemination of career 
information in the district? 

a. High . A definite coordinated effort between the couns^ .ing 

and instructional department exists . 

b. Average . Some career information is disseminated to students. 

c. Low . Very little is being done with regard to dissemination. 

5. Are the Local Plan Objectives measureable? 

a. Yes . They are defir itely stated in measureable tenns. 

b. Almost . The objectives approach measureability but are somev/hat 
global or general in scope. 

c. No . The objectives are not stated in measureable terms. 

6. Does the school have a locally-directed evaluation system? 

a. Yes . A system exists which enables evaluation of the total 

occupational program. 

b. No . An organized system does not ?xist other than testing done 

by teachers or evaluation of staff members conducted by the 
administrators t etc. 



ERIC 



91 



7. If tf yes tf to 6, what groups of individuals are involved in the local 
evaluation system? 

Students 
Teachers 
Administrators 
Counselors 

Advisory commitee members 
Employers 



8. What is the administration's estimate of the percentage of graduates who 
complete four years of college? 

u percent. 



9. What significant changes in program offerings have occurred within the past 
two years? 



10. What is the total number of students enrolled in a cooperative program? 
(D.E., D,0.» 0.0., C.W.T., etc.) 

students. 



11. How would you rata the cooperative programs of the district? 

a. High . A very extensive and comprehensive program is being offered. 

b. Average . A program exists but is not serving all who could benefit 

from such an experience. 

c. Low . A minimal program exists which should be further developed and 

refined. 



12. Is occupational information provided in elementary and other levels of 
feeder schools? 

a. Yes . Most feeder schools are providing occupational information 

to students. 

b. Some . Only some of the feeder schools are providing occupational 

information to students. 

c. No. None of the feeder schools are providing occupational infor- 
mation to students. 



a. 
b. 
c. 
d. 
e. 
f. 
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13. How many years has the administrator in charge of - tcupational education 
held that position within the district? 

years. 

14. How would you rate the effectiveness of existing occupational student 
organizations within the district? 

a. High . Most organizations are effective in student development, 

b. Average . A few of the organizations are effective but most are 

minimally effective. 

c. Low . Organizations are very limited in membership and attendance. 

15. Does the district actually identify disadvantaged and handicapped students 
as stated in the Local Plan? 

a. Yes . 

b. No. 

16. Does the district serve disadvantaged and handicapped students as stated 
ir the Local Plan? 

a. Yes . 

b. No. 

_ c. In some cases. 



17. Are local community personnel involved in planning local programs? 

a. Yes . In most occupational areas. 

b. Limited to only some areas. 

c. No. Very few or none are involved. 



18. Rate the extent to which occupational programs utilize available community 
resources. 

a. High . School personnel utilize community resources extensively. 

b. .Average . School personnel utilize community resources on a 

limited basis. 

c. Low . School personnel do not utilize community resources*. 
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APPEHDIX C 



WORKSHEET 

Local 1-Ytar and 5-Yeer Plans 
for Vocational and Technical Education (19 

SECTION A 

Ganaral Information 



Region. 
19 ) 



.County. 



Legal Name of District. 



Plan Author Name. 



. Dtst. No. . 



, Superintendent. 



Address ft Telephone . 



ECTION B 
Administration Organization 



AVC 



JOINT AGREEMENT 



SECTION C 



District Policies, Goals and Objectives 



1. 



Objectives (Meeeurability 

.A. Proposed Program 
-B. Curriculum Emphasis 
-C. Articulation 

Meeting Student and Community Needs 
.A. Course Descriptions 
-B. Criteria Listing 
-C. Multipliers Claimed 
.D. MDTA 



•Special Contracts 



3. 




5. 



Use of Community Resources 
. A. Resources (I ntemal ft External) 
_B. Resources Facilitate Objectives 
_C. Type, Role, Function of Advisory Council 

What Ancillary Services 
A. Vocational Guidance ft Placement * ' 
, B. Student Organizations 

Evaluation of Proposed Pro yam 
.A. Criterie 
.B. Who Will Evaluate 

•C. Relationship With 1 (Objectives), 2 (Target Groups) 



Special Organization: 



""Initial Program: 



Handicapped: 



9* 



Improve Quality 

. A. Program I mprovement (All) 
.6. Progress Toward Full Approval 

Improve Quality - 5 Years 

.A. Program I nnovrtons 

.8* In-Service 

_C. Facilities 

_D. Joint Agreement 

_E. Area Resources 

_F, Summer Programs 

_G. Elementary Program Approved 

_H. Summer Program 



Disadvantaged: 



SECTION D 



Annual Plan and Approval 



Year 


Industrial 
Oriented 


Health 


Ag. & Applied 
Biol. 


Business 
Mktg. & Mgmt. 


Personal & 
Public Serv. 


Previous 
Year 












This 
Year 













Credit Courses: 



Non-Credit Courses: 



Credit Programs 
Per Credit Cost s $ 



%Credit ■ 



SECTION E 
Long-Range Plan 

SECTION F 

SECTION G 
Summary 

SECTION H 



4 



Non-Credit Programs 



V. E. Reimbursement from State*$ 



% Non-Credit* 



Status of Local Plan 
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APPENDIX D 



Approved 

STATUS OF PLAN 



School Name % Conditionally Approved 

REPORT 



Not Approved. 



District Number 

Enrollments are projected in the following areas: 

' ndustrj j al Agriculture and Business Personal and 
Onented Health Applied Biology Marketing and Management Public Services 

On pages VE-4 and VE6 in column 17 of the Plan those courses that appear to meet the minimum criteria for funding as 
outlined in the State Plan, and according to the State Board policy have been so indicated, using the following categorical 
codes (Numbers are used for identification purposes only) (LETTERS ARE FOR PROGRAM FUNDING PRIORITIES) 

Occupational Training 1 Highest Priority ; - A 

Occupational Related 2 Above Average Priority - B 

Occupational Preparation and Orientation — 3 Average Priority — c 

Occupational Information 4 Below Average Priority Tl_ 0 

Not approved for funding n/a 

Review Comments: 

□ Section C, questions 1 thru 5, have been judged to be acceptable until the district has on-site DVTE evaluation Only 
a duplicate of this portion of Section C with modifications need to be submitted am.ualfy until further notice. Questions 
6 & 7 needs to be revised annually. 



□ 



Section C was judged to be unacceptable for the following reasons:. 



Other Review Comments: 



Additional information is requested on the following to be submitted within ten days of receipt of this mailing: 



improper information may" ^pard.e L^^Z^T^^J,^ ^ °' ~ " 
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APPENDIX F 



SCHOOL AND COMMUNITY DEMOGRAPHIC DATA 



1. Legal Name of your District 



ERIC 



2. Do citizens from your community serve on a district occupational education advisory 
committee? Yes No 

a. If yes, is there: 

one advisor/ committee for all occupational areas, or 

separate committees for each area? (List the separate occupational 

areas having -dvisory committees.) 



b. How many meetings are held each year? i 

c. Are minutes of the meeting(s) available for review by the visitation team 
members? Yes No 

3. Are there active vocational student organizations in the school? 
Yes No 

If yes, list the name of the organizations and the number of students involved in each. 
Name of Organizations Number of Students Involved 



4. What is the current unemployment rate for your area as designated by the Illinois State 
Employment Service? 

5. What is the current youth unemployment rate for your area as designated by the Illinois 
State Employment Service? 
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6. Provide the current unduplicated enrollment for each of the occupational areas by grade 
level. (Make sure that a s f udent is not counted in more than one class.) 




9th Grade 














10th Grade 














11th Grade 














12th Grade 














total: 


3 













What tests*are currently utilized in your school under the following areas? 

Name of testU) When given, and to whom 

a. Interest 

b. Aptitude 



8. 

9. 
10. 



11. 



What was last school year's dropout rate for all high school students? (Dropout rate is 
equal to the percent of all school leavers other than transfers) 

Does your school have a specific counselor designated as vocational? Yes No 

Does your school pr ovide pla cement services for: 



a. Current graduates 

b. Dropouts 

c Former graduates 



Yes_ 
Yes_ 
Yes . 



No 
No 
No 



Does your school have a follow-up program that gives a picture of your entire student 
body? Yes No 



Only vocational students? Yes 
If yes: 



No 



a. How often are follow-up studies conducted? 

b. Who is responsible for coordinating the follow-up study?. 



Are the results of follow-up studies disseminated to the instructional staff? 
Yes No 



ERLC 
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APPENDIX G 
EVALUATION LOG 



FISCAL YEAR: 
COOE NUMBER: 
TEAM LEADER: 



DISTRICT NAME 



TEAM MEMBERS 



ER^C 



TEAM 
MEMBER 
REQUEST 
LETTER 



TEAM MEMBER 
RESPONSE 
(PHONE NO.) 
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VISITATION DATES 



ORIENTATION DATE 



LETTER TO ESR 



LETTER TO DISTRICT 



NOTIFICATION LETTER 



MAILING OF INFORMATION 



FOLLOW-UP LETTER 



REPORT MAILED 



TRAVEL VOUCHER 



Appendix H 



ETS Questionnaire on Profile Ratings 
for Team Members Who Visited Educa- 
tional Agencies in 1971-72 



State of Illinois 



ADVISORY COUNCIL ON VOCATIONAL 6DUCATI0N 



222 SOUTH COLLEGE 
SPRINGFIELD. ILLINOIS 62706 
PHONE 217-528.2892 



Richard B. Ogilvie 

OOVtftNOft 



September 11, 1972 



William E. Nagel 



cxccuTtvr omecTo* 



Dear Visitation Team Member: 

This past year you participated in the on-site evaluation of one or 
more occupational education programs in Illinois . Each visit was ox. 
part of the Three Phase Evaluation System sponsored by the Division 
of Vocational and Technical Education and resulted in an Evaluation 
Report which was submitted to and discussed with the agency you visited. 

The Illinois Advisory Council on Vocational Education, for which I am 
Executive Director, is appointed by the Governor in accordance with 
Public Law 90-576. Its thirty- four members are drawn from business , 
industrial , educational and other fields and is representative of the 
different geographic regions of the State. We are charged with respon- 
sibility to submit an annual evaluation report describing the effec- 
tiveness of federally assisted vocational programs to the U. S. Commis- 
sioner of Education and the National Advisory Council. 



This year we have contracted with Educational Testing Service to help 
us learn hpw effective the Three Phase Evaluation System has been. This 
letter introduces Educational Testing Service. We request that you 
assist them in their work by responding , in accordance with your best 
judgment, to the questionnaire that will accompany this letter. Please 
be assured that your responses will be of real use to us and that your 
individual responses will be kept confidential . 

Thank you for your kind assistance. 




Sincerely 



ERJC »en . P d 
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(Accompanying letter) 



QUESTIONNAIRE ON PROFILE RATINGS 
for 

Tcan Members who Visited Educational Agencies in 1971-72* 



Dear 

Our records show that in you participated in a visit to: 

The Occupational Program Performance Profile that resulted from that visit i 
presented on the following page. I would like to ask you several questions 
about the rating process as well as about the final profile ratings. 



A. 



B. 



How were the ratings derived? Were they primarily 
arrived at through "informal discussions' leading to 
agreement or through a "formal rating" procedure 
involving the pooling of numerical ratings by 
individual team members? (Circle one number) 

Ed. Non-Ed . 

Informal discussions 1 51% 62% 

Formal ratings 2 44% 34Z 

No response " 3 * 

Please consider all of the eight performance profile 
categories as a whole. How do these profile ratines 
compare to the ratings you would have assigned as an 
individual? Overall are the ratings higher, i™". 
or similar to those you would have assigned? (Circle 

onenUB,ber) E<h Non-Ed . 

Higher 1 8% 10% 



C. 



Lower 



7% 



8% 



Similar 3 82% 79% 

No response * A J/ * 

From what you remember, how would your individual ratings 
on each of the eight performance profile categories have 
compared with those presented in the final profile ratings 
on the following page? Mark the attached profile as 
illustrated below by drawing a line through each bar to 
show where your individual rating would have been. 



Administrative Organization 

Personnel 

Objectives 



















fr 






-I 









*148 questionnaires were mailed. Percents are based on the responses < 
the 118 individuals whose questionnaires were, returned by November 1, 1972. 



Tocal Occupational Program Performance Profile 



Administrative Organization 
Higher 1 



Lower 
Similar 
No Response 



Personnel 



Objectives 



2 
3 



Higher 1 

Lower 2 

Similar 3 
No Response 



Higher 1 

Lower 2 

Similar 3 
No Response 

Evaluation 

Higher 1 

Lower 2 

Similar 3 
No Response 

Occupational Programs 

Higher 1 
Lower 2 
Similar 3 
No Response 

Resources Utilized 

Higher 1 
Lower 2 
Similar 3 
No Response 



Educators 

12% 
37 
35 
16 



21% 
32 
32 
16 



11% 
37 
35 
18 



12% 
26 
40 
21 



14% 
30 
39 
18 



18% 
28 
35 
19 



Non-educators 

21% 
38 
16 
24 



8% 
31 
33 
28 



26% 
18 
30 
26 



21 
13 
36 
30 



21% 
25 
30 
25 



16% 
33 
30 
21 



Guidance Services 

Higher 1 
Lower 2 
Similar 3 
No Response 

Students Served 

Higher 1 
Lower 2 
Similar 3 
No Response 



18% 
42 
23 
18 



14% 
30 
37 
20 



11% 
34 
26 
28 



5% 
30 
34 
31 
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D. 



E, 



Do you believe that the agency you visited was making a 
satisfactory effort to: (Circle one number for each item) 

Determine what skills its students need Yes 1 

No 2 



Have up-to-date equipment for its 
instructional program? 



Develop a work-study program' 



Place its students in jobs? 



Follow up on its students after they 
left school? 



Determine what skills its program 
graduates possessed? 



Have a sound working relationship 
with persons like yourself in its 
service area? 



No Response 

Yes 1 
No 2 
No Response 

Yes 1 
No 2 
No Response 

Yes 1 
No 2 
No Response 



Yes 1 
No 2 
No Response 



Yes 1 
No 2 
No Response 

Yes 1 
No 2 
No Response 



As you think back to your visit, would 
you say that you were positively impressed 
by what you learned about the programs of 
the agency? (Circle one number) 



Yes 



1 

2 



F. Do you feel that your training to be 
a team member was adequate? 
(Circle one number) 



No Response 



Yes 1 
No 2 
No Response 



Ed* Non-Ed. 



12 



89% 
9 
2 



53% 


70% 


46 


28 


2 


2 


75% 


85% 


25 


13 




2 


66% 


72% 


28 


25 


5 


3 


51% 


51% 


46 


46 


4 


3 


32% 


43% 


61 


56% 


7 


2 


35% 


43% 


58 


49 


7 


8 


65% 


61% 


26 


36 


9 


3 


49% 


77% 


39 


20 



79% 
21 



9 

ERIC 



G. Do you believe that visitation teams 
of five or more individuals should 
include at least two persons who 
are neither educators or students? 
(Circle one number) 
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Yes 1 
No 2 
No Response 



86% 89% 
14 10 
2 



H. 




Non-Ed . 
79% 



No 2 2 

No Response 12 
Are you positively impressed by the operation of the 
Three Phase Evaluation System? (Circle one uumber} — 



13 
8 



I. 



Yes 



1 



84% 



89% 



No 2 7 

No Response 9 
What is the exact title of your present occupation? 



7 

5 



J. 



I appreciate the time and attention you have given to these questions. 
If you have any suggestions you would like to make for the improvement of the 
Three Phase Evaluation System, they may be recorded in the space just below 
my signature. Please return the questionnaire in the self-addressed, stamped 
envelope which has been enclosed. Thank you again for your cooperation. 



Sincerely yours, 




Daniel P. Norton 

Director of Research, Midv;estern Office 
Educational Testing Service 
Evanston, Illinois 60201 
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Appendix I 

ETS Questionnaire for 
Program Planners 



State of Illinois 

ADVISORY COUNCIL ON VOCATIONAL 6DUCATI0N 



222 SO v jLLECE 
SPRINGFIELD, ILLINOIS 62706 
PHONE 217-929*2892 

Richard Ogilvie „ ^ _ , , , William E. Nagcl 

oovknnor September 11, 1972 cxccunvc omccto* 



Dear Occupational Program Planner : 

Your agency has submitted a plan for occupational education to the 
Illinois Division of Vocational and Technical Education for Fiscal 
1972-73, of which you have been listed as the preparer. I an writing 
to you in your role as a planner and an individual who is highly 
informed about planning and evaluation as they are undertaken by 
local education zigcr.ai cs . 

The Illinois Advisory Council on Vocational Education , for which I am 
Executive Director , is appointed by the Governor in accordance with 
Public Law 90-576. Its thirty- four members are drawn from business, 
industrial and other fields and is representative of, the different 
geographic regions of the State. We are charged with responsibility 
to submit an annual evaluation report describing the effectiveness of 
federally assisted vocational programs to the U. S. Commissioner of 
Education and the National Advisory Council. 

This year we have contracted with Educational Testing Service to help 
us learn how effective the Three Phase Evaluation System has been. 
This letter introduces Educational Testing Service. We request that 
you assist them in their work by responding , in accordance with your 
bes t judgment , to the questionnaire that will accompany this letter. 
Please be assured that your responses will be of real use to us, and 
that your individual responses will be kept confidential . 

Thank you for your kind assistance. 



WEti . 
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Dear Mr. 

As the administrator who was primarily responsible for the preparation 
of the F.Y. 1971-72 planning proposal submitted by your agency to the Illinois 
Division of Vocational and Technical Education (DVTE), you can be of particular 
f ielp in the Council sponsored assessment of the effectiveness of the DVTE Three 
Phase System for Planning and Evaluation. Your answers to the questionnaire 
below will be used by the Council for guidance and recommendations on future 
plans? - 

All individual questionnaires will" be treated as confidential mate ial, 
and only summary analyses of pooled data will be delivered to the Council. 
Please be entirely frank in your answers, and return the completed question- 
naire iji the enclosed envelope as soon as possible. 

Sincerely yours, 



Daniel P. Norton 
Director of Research, Midwestern Office 
Educational Testing Service 
Evans ton, Illinios 60201 



THREE PHASE SYSTEM QUESTIONNAIRE 
for Program Planners 



Please circle one of the letters following each statement that best describes 
your response. 

SA - Strongly Agree 

A - Aoree 
%i 

D - Disagree 
SD - Strongly Disagree 
NA - Not Applicable or No Opinion 

No 
Response 

(0) (1) (2) (3) (4) (5) 



1. The forms which the DVTE provides to local agencies 
are well designed and the instructions are clear 



and easily understood. 




SA 


A 


D 


SD 


NA 




% 5 


5 


49 


28 


12 




I know the proposals I prepare will be reviewed 














fairly and on schedule. 




SA 


A 


D 


SD 


NA 




% 3 


16 


65 


11 


4 


2 



0 

ERIC 



*137 questionnaires were mailed. Percents are based on the responses of the 
114 individuals whose questionnaires were returned by November 1, 1972. 
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No 
Response 



ERIC 





(0) 




(1) 


(2) 


(3) 


(4) 


(5) 


3. 


I understand the criteria by which our plans are 


















reviewed and judged. 






SA 


A 


27 


527 


2IW 




% 


2 


8 


55 


32 


2 


1 


4. 


We have gained a lot in our planning efforts from 
















having employer representatives advising us. 






SA 


A 


27 


527 


NA 




% 


3 


14 


48 


18 


8 


9 


5. 


The DVTE is doing a good job of helping our agency 
















to understand how to meet financial accountability 
















requirements. 






SA 


A 


27 


527 


27/1 




% 


2 


4 


31 


50 


8 


5 


6. 


Our planning activities would have been more 


















successful with external help from the DVTE or 


















some other agency to : 


• 














a. Conduct student follow-up studies. 






SA 


A 


27 


527 


NA 






% 


4 


12 


45 


26 


9 


4 




b. Conduct need survey in our community. 






SA 


A 


27 


527 


27/1 






% 


5 


15 


40 


29 


7 


4 




c. Conduct employer need surveys. 






SA 


A 


27 


527 


27/1 






% 


5 


11 


45 


29 


6 


4 




d. Conduct student interest surveys. 






SA 


A 


27 


527" 


27/1 






% 


5 


11 


31 


41 


10 


3 




e. Prepare behavioral objectives. 






SA 


A 


5 


527 


27/1 






% 


5 


24 


37 


25 


7 


3 




f. Conduct effective evaluations. 






SA 


A 


27 


527 


NA 






% 


6 


12 


56 


18 


5 


3 




g. Identify job entry skills required of students. 




SA 


A 


27 


527 


27/1 






% 


5 


18 


44 


25 


5 


3 




h. Prepare comprehensive planning documents. 






SA 


A 


27 


527 


27/1 






% 


5 


16 


42 


26 


5 


5 




i. Set priorities for program development. 






SA 


A 


27 


527 


NA 






% 


5 


14 


39 


29 


9 


4 




j. Organize advisory committees. 






SA 


A 


27 


527 


27/1 






% 


4 


5 


22 


53 


13 


3 


7. 

9 * 


Schools that are to be visited this year ('72-' 73) 
















will benefit from the preparation they will make 


















for these visits. 






SA 


A 


27 


527 


NA 






% 


2 


32 


48 


12 


2 


4 


8. 


Visitation team members should be drawn from more 


















than 50 miles for the agency they will visit. 






SA 


A 


D 


527 


27/1 




% 


4 


•18 


29 


34 


9 


.7 


9. 


Less effort was necessary to prepare a plan 


















for 1972-73 than to prepare one for 1971-72. 






SA 


A 


27 


527 


274 






% 


3 


14 


36 


28 


14 


4 


10. 


The Planning and Evaluation Unit of the DVTE 


















tries to be helpful to agencies when they 


















are preparing their plans. 






SA 


A 


27 


527 


274 






% 


2 


18 


62 


9 


1 


9 


11. 


It is difficult to know exactly which students 


















should be classified as handicapped. 






SA 


A 


27 


527 


27/1 






% 


3 


21 


47 


22 


5 


2 
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No 












- ^ 

k i 






Response 


















(0) 


(1) 


(2) 


(3) 


(4) 


(5) 






12. 


The amount of money we receive for our programs 






- 








i: 






does not justify the cost of our planning effort. 




SA 


A 


D 


SD 


NA 








% 


7 


20 


25 


33 


12 


3 






13. 


Students have been a big help to us in our 




















curriculum development. 




SA 


A 


D 


SD 


NA 


*■ 






% 


2 


4 


52 


39 


2 


3 


1 j 




14. 


Our counselors are well qualified to do 




















occupational counseling. 




SA 


A 


D 


SD 


AM 


f 






% 


3 


11 


46 


30 


9 


2 






15. 


We believe we should formally help our graduates 




















and dropouts get placed in jobs. 




SA 


A 


D 


SD 


Ml 




* 




% 


4 


26 


46 


15 


4 


4 






16. 


I would like to see our programs judged by the 




















occupational skills our students acquire. 




SA 


A 


D 


SD 


Ml 


I J 






% 


3 


13 


61 


16 


2 


5 


* 




17. 


My DVTE Regional Director has given me a great 




















amount of help in my efforts to plan well. 




SA 


A 


D 


SD 


JIM 


1 - 5 


- 




% 


2 


27 


46 


18 


3 


5 


5 | 




18. 


Employers do not know enough about the skills 














: * 






their employees need for us to develop programs 




















closely related to their needs. 




SA 


A 


D 


SD 


AM 








% 


4 


5 


23 


48 


12 


7 






19. 


It is not important to have fully modern 




















equipment for occupational programs. 




SA 


A 


D 


SD' 


AM 








% 


4 


6 


25 


33 


31 


1 






20. 


Site visitation will prove to be a waste of time 
























SA 


A 


D 


SD 


AM 








% 


3 


3 


13 


46 


32 


4 






21. 


It is easy to involve advisory committee 




















members in planning. 




SA 


A 


D 


SD 


NA 








% 


3 


4 


25 


48 


20 


1 






22. 


The guidelines we received helped us to develop 




















needed new programs. 




SA 


A 


D 


SD 


NA 








% 


3 


2 


45 


32 


11 


7 






Please respond to these items only if you were visited last year 


• 












23. 


Our visitation team prepared a fair and 




















competent report. 




SA 


A 


D 


SD 


NA 






24. 


% 


1 


13 


25 


6 


4 


mm 






We knew what we had to improve upon to have 
















this year's program plans approved. 




SA 


A 


D 


SD 


NA 






25. 


% 


1 


9 


31 


7 


1 


1 






The team members who came to our agency were 




















well prepared to do their work. 




SA 


A 


D 


SD 


NA 






26. 


% 


1 


10 


28 


9 


2 


— 






We received good followup support from the 
















■ » 




DVTE after we were visited. 




SA 


A 


D 


SD 


NA 






27. 


% 


1 


5 


25 


11 


4 


2 






The visitation phase of the Three Phase System 




















should be continued at least until each agency 




















has had two visits. 




SA 


A 


D 


SD 


:m 




X'- 




% 


1 


15 


24 


6 


2 


2 




o 




Please provide any additional comments or suggestions on the back of 


t&is pa#e. 




ERIC 
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